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Foreword 



Goodwill Industries of America, Inc. and the University of Wisconsin- 
Stout have entered into an agreement to jointly create publications that 
help improve service delivery in rehabilitation facilities. Manuals developed 
to supplement in-service training programs by the staff of Goodwill will be 
further developed by Materials Development Center for use by professionals 
in rehabilitation facilities nationwide. 

Program Evaluation: A Self-Studv Manual is based on two manuals 
developed by Goodwill as part of their staff training series on program 
review. The original manuals were titled M A Manual on Program Evaluation' 1 
and "Program Evaluation Exercise Workbook." In this publication, these 
manuals have been combined, edited, and expanded to provide a manual 
applicable to all facilities. The use of program evaluation, explained in this 
manual, will help facility administrators implement and improve their service 
delivery. 



Kenneth J. Shaw, Director Paul R. Hoffman, Acting Dean 

Rehabilitation Services School of Education & Human Services 

Goodwill Industries of America University of Wisconsin-Stout 



Preface 



The focus of this manual is on the development of a practical and 
flexible data collection system to support facility program evaluation out- 
come and research models. Four basic criteria must be followed to maintain 
consistency between facility data collection efforts and to maintain integrity 
to the data collection system. They are: 



1. Program evaluation measures what happens to all clients. 
The data collection system must track successful and unsuc- 
cessful clients. 



2. Program evaluation includes outcome and research models to 
measure outcomes achieved following cessation of services 
and other specific evaluation questions. (An outcome model 
measures what happens after a client has completed a pro- 
gram.) 



3. Reports should be produced on a continuous basis, providing 
quick feedback to administrators on the state of their pro- 
grams. 



4. The program evaluation report tells the facility administrator 
if program performance was acceptable according to the 
standard which the board has set. Therefore, each program 
must have predetermined standards for outcomes against 
which actual results can be compared. 
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vide a "self-test" to learners, and the documents formed during the exer- 
cises may actually form the basis for program evaluation within the facility. 



Reference Materials Provided 

For learners who wish to gain more expertise in program evaluation, 
reference materials are provided in Appendix B. 



Workshops 

This manual is designed to provide the core of a self-directed (pro- 
grammed learning) in-service. It may also be used to provide "workshop" 
training. The workshop leader may assign sections for individual learners 
to read, or assign the exercises as group projects. 

Workshop leaders should tailor their presentation to the professional 
learners they will be instructing. They should select methods that draw on 
the experience of their learners, promote efficiency and effectiveness of 
learning, and encourage group participation. Leaders are encouraged to 
read Developing Effective In-service Training Programs (Smith, 1984) prior 
to modifying this guide for use with their classes. It is available from the 
Materials Development Center, University of Wisconsin-Stout. 
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Introduction 



Good management depends upon good data. To improve the quality of 
services, identify service needs, develop new services, and provide services 
efficiently and effectively, facility administrators need timely, accurate and 
concise data that describe the successes and/or failures of their programs. 



Program Evaluation Defined 

Patton (1982) provided a comprehensive definition of program evalua- 
tion. He defined program evaluation as the systematic collection of infor- 
mation about the activities, characteristics, and outcomes of programs and 
services that helped administrators reduce uncertainty in the decisions they 
made regarding the activities, characteristics, and outcomes of the programs 
and services that they administered. Thus, program evaluation is a planned 
process of gathering and analyzing data to help make decisions less risky. 



Practical Program Evaluation 

The above definition "leaves a lot of room for interpretation. It does 
not state how much data should be collected or what data should be col- 
lected, only that the data be collected systematically to help reduce uncer- 
tainty in decision making. Even these areas are loosely defined. 

The biggest danger facing facility administrators is the tendency to 
think that bigger is necessarily better. More data gathered more frequently 
on more objective level may not make a significantly more effective pro- 
gram evaluation. Be practical. Determine evaluation questions, set data 
collection procedures, and choose analysis techniques that provide adequate 
levels of confidence but do not burden staff with unnecessary data collec- 
tion tasks. 



. -14 



Why Bother with Program Evaluation? 



Several general reasons for conducting program evaluation were out- 
lined by Cook and Cooper (1978). They are: Vindication, Salesmanship, 
Verification, Improvement, and Understanding. 



Vindication 

Sometimes program evaluation is conducted to gather facts that show 
that the facility's programs are worthwhile. Thus evaluation hopefully 
justifies the work (and existence) of the facility. 



Salesmanship 

Evaluation is also performed to determine areas that can be expanded. 
The purpose of the evaluation is to find program components that have 
similar applications to potential clientele or to find similar need areas from 
potential client groups that can be met by existing programs. 



Verification 

Verification is another reason for conducting program evaluation. 
Facility administrators often need to discover if the stated goals of a 
program are actually being met. Additionally, the determination of program 
impact, effectiveness, and efficiency are essential needs for administrators 
that can be, at least partially, met through the application of program 
evaluation. 



Improvement 

The analysis of program evaluation data can be a great tool in the 
development of more effective and efficient program systems. Through 
program evaluation, system weaknesses are minimized and strengths are 
maximized. 



Understanding 

Finally, facility administrators use program evaluation to seek an 
understanding of how their programs function. Specific features of a 
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program may be evaluated as an administrator searches for reasons that a 
program is a success or a failure. 



Uses for Program Evaluation 



The facility's program evaluation system may be used for many pur- 
poses. They include: 



The data can be used to increase benefits provided to 
persons served. 

The data can be used to identify the need for new 
programs. 

The data can be used to contain costs. 

The data can be used as a marketing tool. 

The data can be used to strengthen organizational 
relationships. 

The data can be used to improve board relationships. 



Increase Client Benefits 

The data which is collected for program evaluation will allow ad- 
ministrators to better select those persons who have the need for services 
and who can benefit from service provision. Client benefits should increase 
when program evaluation is regularly provided because staff will be en- 
couraged to strive for positive results. Program evaluation systems also 
allow specialized studies to be undertaken resulting in new benefits. 



Develop or Expand Programs 

The data collected will help identify new program or service needs and 
point the direction for the expansion of facility services. Program evalua- 
tion will reduce the risk of new program failure by providing a tool with 
which administrators can structure, plan, and monitor results. 

Facilities with limited resources to invest can establish new programs 
utilizing zero-based budgeting and sunset strategies to determine, in the 
short run or by a fixed period, whether the program is showing positive 
results. Program evaluation can provide data that indicates if the extended 




outlook for a new program looks good enough to invest resources necessary 
for its continuation. 

Program evaluation also helps administrators replicate highly successful 
programs developed by others. 



Contain Costs 

Cost and efficiency objectives and measures in a program evaluation 
system provide essential feedback on the cost of the program in relation to 
client benefits. This information helps administrators determine if costs are 
acceptable compared to other program alternatives. These measures also 
encourage the minimizing of costs, and allow administrators to demonstrate 
to referral and funding sources exactly how much services cost, what areas 
are the most costly, whether the resources provided by the purchasers are 
sufficient to cover costs, and to establish any need for subsidized support 
by the community. 

Decisions regarding the continuation, expansion, or termination of 
programs can most easily and accurately be made when this data is avail- 
able. Decisions as to whether to provide services directly or to contract 
for services can also be made with greater confidence. 



Provide Data for Market-Based Planning 

- Programs can be confidently sold to consumers using program evalua- 
tion data to demonstrate the actual results that can be achieved by the 
program. An enhanced ability to cite the program's record of providing 
services that satisfy specific needs and problems, or to serve special popu- 
lations of disabled clients, will enhance marketing efforts. Program evalua- 
tion may also help administrators identify potential markets for existing or 
new programs. Data gathered from the system should provide information 
regarding new funding sources. 



Strengthen Interactions with Other Organisations 

Many accreditation commissions as well as regulatory agencies require 
program evaluation. Community funding agencies, such as United Way, who 
allocate their funds among many "worthy" causes respond favorably to solid 
information from program evaluation sources. These data provide the 
essential information needed by such agencies to determine their funding 
levels. 

The facility's ability to show success is an asset for establishing and 
maintaining mutually beneficial relationships with consumer groups and other 
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organizations. Additionally, cooperative endeavors and working contract 
agreements are easier to obtain and mainvain if program evaluation data is 
available. Cooperative projects involve joint planning. Maximum use of 
planning resources can be enhanced by the use of program evaluation data. 



Provide Decision-Making Data for the Facility Board 

The forced structure, format, and objective setting of a program 
evaluation system will help board members better understand the facility 
and its programs. Information gleaned from the evaluation data will help 
improve policy making and long range planning by the board. 

The availability of proven statistics about programs, their costs and 
needs, will enhance the ability of the board to mount successful fund 
raising efforts. Regular management reports keep the board up-to-date and 
informed and make administrative presentations easier and more profes- 
sional. 

The use of program evaluation data is a matter of commitment. The 
Board, the Chief Executive officer, and all facility staff must collaborate to 
make ev&luations useful. Commitment to program evaluation must start at 
the top and filter throughout the organization. 

Because facilities exist to provide the best possible services, programs 
are never as good as they "could be." Continuous support is needed to 
improve the programs and to develop other programs and services should 
the need be identified. As part of the program evaluation process, data is 
compiled that will indicate the extent to which programs are effective and 
efficient. Using these data, administrators must develop methods, tech- 
niques, and plans to improve facility performance in the areas where pro- 
gram evaluation indicates that improvement is needed. 

Staff, starting with the CEO, must be held responsible and accountable 
for results. Implementation of a program evaluation system requires com- 
mitment from every level within the organization. It is not sufficient that 
the CEO and staff recognize the need for program evaluation. The govern- 
ing body should also go on record as desiring program evaluation by requir- 
ing its implementation and providing dollar support. Program evaluation, 
like any major undertaking in a facility, requires the expenditure of time 
and resources. 



The Evolution of Program Evaluation 

The use of program evaluation in rehabilitation settings has evolved 
over several decades. Beginning in the 1930's, activities in three areas 
shaped the face of program evaluation as it is practiced today. These three 
areas, applied action research, improved management monitoring techniques, 



and awareness of political needs, have influenced both administrators and 
funding personnel and have provided the theoretical base and technical 
methods required to support evaluation research. 



Applied Action Research 

Experiments in social problems begun in the 1930's and spurred by 
wartime interests in the 1940's led to the development of scientific research 
techniques for investigating social activities. These techniques have evolved 
into data gathering and analysis processes that have a significant impact on 
the way rehabilitation programs are evaluated. Of particular use are the 
concepts of random sampling and the testing of means and percentages. 



Improved Management Monitoring Techniques 

Ever since the Westinghouse studies of productivity in the 1950's, 
administrators have investigated management related questions with ever 
more powerful and sophisticated methods. Their efforts at monitoring 
worker output have been given a great boost by the explosion in the com- 
puter industry. Even extremely small companies using personal computers 
can collate, store, and analyze large amounts of management related data at 
relatively low cost. The primary restricting factor today is the time needed 
to gather the data and enter it into the statistical records of the computer. 



Awareness of Political Needs 

Within the rehabilitation field, probably the most significant impetus to 
engage in program evaluation was given by a changing political climate that 
desired and then demanded accountability as a prerequisite for continued 
dollar support. This awareness that an accurate means for demonstrating 
program effectiveness and efficiency was needed led first to the creation of 
facility standards and then to a proliferation of systems. 



Developing a Program Evaluation Policy 

The purpose then, of establishing a program evaluation system for 
facilities is to clearly determine what services are provided, who receives 
the service, what the services intend to achieve, how successful they are at 
achieving their missions, and what changes may be needed to improve 
service performances* 
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Many uses and benefits should come out of the development of the 
system. Most important is responding to the increasing demand for ac- 
countability by the general public* This demand is reflected in the man- 
dates of referral and funding sources. Program evaluation should also help 
the facility better describe and market its services, with the net result of 
an enhanced public image, increased community support, referrals, and 
funding. 

To evaluate the facility's programs, an evaluator needs direction. Like 
a road map, the facility's program evaluation policy should give the evalu- 
ator a guide to the "paved roads." The evaluator needs to use the policy 
to determine the approach to use, specific tactics or methods to use within 
the approach, and the criteria or standard upon which the evaluation is 
based. 

Determining a plan of attack for every evaluation that is performed 
can be a time consuming and chaotic task. Cook and Cooper (1978) state 
that program evaluations can use different criteria to reach a conclusion 
about a program's value: 

The most logical criterion of worth is the program's effec- 
tiveness or performance in meeting stated goals. The emphasis 
here is on impact or consequences. A program which has as its 
goals various operations may be able to show that it was effec- 
tive in completing those operations. However, unless there is 
some measure of output or reason for the program; existence 
beyond mere performance, the evaluation will be of little ultimate 
value. Other criteria of program worth give attention to the 
process underlying the ultimate outcomes. One process related 
criterion is effort which is a measure of program inputs or 
resources expended. Closely related to effort is efficiency which 
Bennet and Weisinger (1974, p. 4) define as "a rate that is pro- 
duced when the effort expended by a program is compared to the 
program's effectiveness." Efficiency is different from effective- 
ness. For example, Nazi Germany was seen, in an industrial/or- 
ganizational sense, as being very efficient, but from their point 
of view, the ultimate outcome was not very effective, (p. 15) 

Thus, it is essential that facility administrators develop a policy that 
will guide the development of program evaluation and give direction to the 
implementation of an ongoing evaluation process. The policy should be 
written and reviewed periodically to insure that the facility's program 
evaluation system is gathering needed data and that those charged with 
specific evaluation activities are carrying out their duties. Figure One is a 
sample of a facility program evaluation policy. It contains the usual ele- 
ments assigning tasks and roles to administrative staff, and gives directions 
for the carrying out of the policy. 

7 
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FIGURE ONE: A Sample Policy for Program Evaluation 



Elk Mound Integrated Industries 

Policy: An organized program evaluation system covering all programs 
operated by Elk Mound Integrated Industries (EMII) shall be developed by 
the EMII Executive Director under the instruction of the EMII Board. This 
policy is effective on May 19, 1986. The program evaluation system must 
be in effect by January 1, 1987, and will be revised yearly at the annual 
board meeting as needs dictate. 

Purpose: The purpose of the program evaluation system is to provide the 
EMII Board and staff, and the clients and taxpayers supporting EMII data 
that identifies the effectiveness and efficiency with which EMII provides 
services, and to determine ways in which this performance can be improved. 

Responsibility: EMII Executive Director, Robert Bell, is responsible for all 
aspects of the program evaluation system development and operation. 

Authority: EMII Executive Director, Robert Bell, or his designee(s) are 
empowered with all authority reasonably necessary to undertake a com- 
prehensive program evaluation and to ensure that such an effort is main- 
tained. 

Monitoring and Review: The monitoring and review functions will be 
carried out by EMII staff as assigned, on a semiannual basis. The results 
of these reviews will be reported to the Board semiannually. 

Reports: The EMII Executive Director will report to the Board the program 
evaluation results. These results will compare expectations as outlined in 
establishment criteria with actual data. 

Kev Roles: 

Board of Directors 

1. Review the management report and the essential aspects of 
the program evaluation system. 

2. Clearly articulate to all EMII administration the importance 
attached to this program evaluation effort and the expecta- 
tion that cooperative efforts of all EMII staff will be made 
to insure satisfactory completion of the effort. 

3. Review the management report to monitor actual performance 
as compared to established criteria and policy fulfillment. 

8 
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Executive Director 



1. Install a program evaluation system by the target date. 

2. Maintain the evaluation activities and communicate results to 
the Board. 



Executive Director and/or Designee 

1. Implement a systematic and continuous program evaluation 
system to determine the effectiveness and efficiency with 
which results are achieved by the staff and systems of EMII. 

2. Develop the program evaluation system using standardized 
guidelines as outlined by the Commission on Accreditation of 
Rehabilitation Facilities (CARF). 

3. Provide leadership in the implementation phase of the instal- 
lation of the program evaluation system. 

4. Provide leadership in the anaiysis and reporting phases of the 
program evaluation system. 

5. Ensure the accuracy of the data collected in the program 
evaluation process and establish systematic reporting pro- 
cedures for the evaluation system. 

6. Compile all results into a single management report, append- 
ing an analysis of the data, in preparation for filing with the 
Board. 

7. Monitor and evaluate the status of the program evaluation 
system at least quarterly to insure that the system and the 
data collected remain relevant to the needs of EMII ad- 
ministration. 



Program Directors and Staff 

1. Develop data tracking mechanisms for all measures identified 
for the system by the Executive Director and/or Designee. 

2. Compile accurate data and report the data on monthly man- 
agement reports. 

3. Make appropriate recommendations to improve EMII services 
and the EMII program evaluation system. 
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Adopted by the board as Policy # 3-86 

Date: 

Recorded: 
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EXERCISE ONE: Develop a Program Evaluation Policy 



Using the model above, create a program evaluation policy for your 
facility. The policy you create may be used in the design of a system for 
your programs, so be thorough, and create a policy that "fits" with present 
operating procedures in your facility. Your policy should include the fol- 
lowing elements: 

1. Assignment of program evaluation responsibility. 

2. General purpose of the evaluation. 

3. An indication of where data about the approach to 
be used and the measurement criteria will be 
found. 

4. Roles of key personnel. 
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SELF-TEST: Introduction 



1. Good management depends on good . 

2. Program evaluation is the collection of information about 

the activities, characteristics, and outcomes of 

and that help administrators in 

the decisions they make. 

3. When designing your program evaluation, be , 

4. Five reasons for conducting program evaluation are: , 

, * » and . 

5. Program evaluation data can be used to: 



and . 

6. Three areas have shaped the way program evaluation is conducted 

today. They are: . , > 

and . 

7. A program evaluation policy provides a to the program 

evaluation. 

8. A program eva<uat;.on policy is needed because many criteria may be 

used to determine program worth, including: , 

and , 
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SELF-TEST: Introduction 
ANSWER KEY 



1. Good management depends on good data. 

2. Program evaluation is the systematic collection of information about the 
activities, characteristics, and outcomes of programs and services that 
help administrators reduce uncertainty in the decisions they make. 

3. When designing your program evaluation, be practical. 

Five reasons for conducting program evaluation are: vindication, sales- 
manship, verification, improvement, and u n derstanding, 

5. Program evaluation data can be used to: increase client benefits. 
develop or exna nd programs, contain costs, develop market-based plans. 
strengthen in fractions with other organizations, and provide decision 
making data. 

6. Three areas have shaped the way program evaluation is conducted 
today. They arc: applied action research, improved management moni- 
toring techniques', and awareness of political needs. 

7. A program evaluation policy provides a guide to the program evaluation. 

8. A program evaluation policy is needed because many criteria may be 
used to determine program worth, including: e ffectiveness, effort, and 
efficiency. 
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Chapter One 
Types of Program Evaluation 



Program Evaluation Answers Questions 

Evaluations are conducted for many reasons, but all the reasons have 
one thing in common: they are questions about the operation of programs. 
The questions initially may be vague, such as: What are our needs? What 
does our system look like from an outcome point of view? Are we meeting 
our goals? It is the evaluator's role to clarify the questions and determine 
the best evaluation methods to use to give the desired answers. 



Select an Evaluation to Answer Specific Questions 

When a facility board tells the administrator that they want a program 
evaluation, they will really be asking the administrator to answer questions 
in one of five different areas. These areas are: 

Needs assessment 

A description of the program in operation 
Measurements of goal achievements 
Continued and smooth program operation 
Determination of the program's fate 

(Morris and Fitz-Gibbon, 1978) 

The deciphering of a request to determine an evaluation need is 
usually left up to an evaluation specialist. The specialist, however, must 
never proceed with an evaluation without consulting with the question 
givers, giving an explanation of the evaluation to be performed, the ques- 
tions to be asked, and the expected results. In this manner, the evaluator 
affirms that desired information is being obtained and that all of the needs 
of the question givers are being met. 
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The evaluation specialist has three different types of evaluations to 
choose from when determining the approach to use. They are: formative 
evaluations, summative evaluations, and needs assessment. In addition, the 
evaluator may use one of many different models of program evaluation to 
obtain answers to evaluation questions. The models are variations of two 
progran evaluation approaches: the objective attainment approach, and the 
systems analysis approach. 

In all cases, data to support an evaluation is obtained from research. 
The investigative research that is conducted utilizes one of three broad 
tactics: experimental research, correlational research, and case studies. 



Formative Evaluations Determine Future Activities 

Formative evaluations are developed to answer broad questions such as: 
How can the program be improved? How can it become more effective and 
efficient? Thus, a formative program evaluation makes use of evaluation 
data to further develop the program. These evaluations tend to be ongoing 
in nature. This is the type of evaluation that CARF standards require, 
because, after the system is in place, data that describe and the status of a 
program are continually obtained. 

Effort and efficiency are important to consider in a formative program 
evaluation. Once the data collection process has been implemented, staff 
time will be committed to a long term gathering process. Thus, in a for : 
mative evaluation it is important that program administrators formulate the 
most efficient ways to meet their stated goals. 

Formative evaluations usually require the collection of a large amount 
of diverse data. To improve a program, it is necessary to understand how 
well a program is moving toward its objectives so that changes can be made 
in the program's components. It is also necessary to identify and gather 
data about the hundreds of tasks that are performed within the program 
system. Thus, formative evaluation is time consuming. 



Summative Evaluations Describe Present Status 

In a summative program evaluation, the emphasis is on program effec- 
tiveness or outcome and the adequacy of program performance or quality. 
Its task is to make "summary" judgments about the program and its value 
usually leading to decisions about the continued operation of the program. 

To make judgments using a summative evaluation, it is necessary to 
compare the data gathered during the evaluation with another program 
aimed at achieving similar goals. If no rival program is available, the 
evaluation data could be compared with a group of individuals identical to 
the individuals in the program, but who did not participate in the program. 
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Summative evaluations are usually terminal or "one shot" efforts 
mounted to answer questions such as: Is Training Program A worth con- 
tinuing or expanding? How effective is Training Program A? and What 
conclusions can be made about Training Program A? Summative evaluations 
are also titled outcome evaluations, consumer testing, or evaluation re- 
search. 



Needs Assessment Measures Desires 

Many requests for program evaluation actually require a needs assess- 
ment. Questions such as: What should the program try to accomplish? 
What areas of the program need attention? and Where is the program 
failing? are really questions regarding the discovery of weaknesses or 
problem areas in the current program. These questions are best investi- 
gated by a needs assessment rather than a formative or summative program 
evaluation because the resulting report is used for long term planning 
rather than immediate change. Requests for "evaluation" within large, 
complex programs are often really requests for needs assessment. See 
Smith, C. (1984) Developing Effective In-service Training Programs for a 
discussion of needs assessment techniques. 



Objective Evaluations Measure Criterion Success 

Many vocational rehabilitation facilities routinely use the objective, or 
goal, attainment approach to determine if their clients are making progress 
within training programs. They often use this approach for the evaluation 
of facility programs as well. Using the objective attainment approach, 
facility administrators ask if a program has accomplished what it was 
designed to accomplish. 

The logic behind the approach is sound; the program is evaluated on 
the basis of meeting predetermined objectives. In practice, however, the 
application of the goal attainment approach is not as cut and dried. Be- 
cause program goals and objectives are often set in reaction to value 
judgments, disagreement can arise as to the "real" value of the goals and/or 
objectives. If the value of the goals and objectives are in dispute, the 
value of data showing that the goals and objectives were met is also dis- 
puted. In fact, even if a goal or objective is perceived by many as valu- 
able, clearly stating the terms and measures that describe "success" may be 
difficult. 

In addition, it is important to emphasize that a program evaluation 
based on an attainment approach will not be able to clearly state that the 
program alone is responsible for the attainment of the goals and objectives. 
The evaluation will not be able to show if another program would have 
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been more effective or efficient. Goal and objective achievement does not, 
by itself, give any basis for judging comparative benefits of one program 
with another. 



Systems Analysis Considers Activities and Flow 

The systems analysis approach focuses on the analysis of functional 
organizational units to arrive at a judgment of program value. The effec- 
tiveness of a program, thus, is not determined only in relationship to stated 
goals, but also in terms of its contribution, to the functional operation of 
the entire facility. 

The systems analysis approach to program evaluation relies on the 
gathering of very large amounts of data. Thus, the popularity of the 
approach has grown with the proliferation of personal computers and the 
trend toward gathering large amounts of data for "accountability" purposes. 

Data accumulated for program evaluation using the systems analysis 
approach is useful, in sophisticated programs, for the projection of out- 
comes under varying conditions. Thus, systems analysis allows program 
evaluation to be predictive, providing a valuable planning tool to facility 
administrators. 

This approach is most useful for evaluations that examine production 
and management issues. It is less useful when the data to be examined is 
intangible, such as an evaluation to determine staff morale or client "hap- 
piness." 



Cost Effectiveness Measures Cost Per Outcome 

The basic idea of the cost effectiveness tactic in program evaluation is 
to associate each positive outcome of a program with the costs associated 
with obtaining the outcome. Cost/benefit analysis is the resulting equation. 

"Cost effectiveness" of program services, stated in terms of specific 
objectives, will provide facility administrators with financial data by which 
to: 

1. Communicate to referral agencies and public its actual costs 
for specific rehabilitation goals. 

2. Substantiate expenses which are not being met by govern- 
mental sources and, therefore, must be subsidized. (This can 
be important data useful for requesting additional community 
funds through the united way, local government, fund raising 
campaigns, etc.) 

3. Negotiate fair fees-for-service. 

4. Maintain cost containment. 
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5. Make management decisions regarding the maintenance, de- 
crease, or increase of a particular program and its associated 
costs. 



To provide cost/effectiveness evaluations, one positive objective must 
be stipulated for each program though the objective is usually composed of 
several programmatic activities. Thus, the data in some programs may be 
added with data from other programs to obtain the total number of suc- 
cessful outcomes. 

A special problem exists in regard to work activities, sheltered em- 
ployment, and supervised residence programs. Due to the small numbers of 
clients from these programs who obtain competitive or independent situa- 
tions (and thereby meet the objective characteristics for successful closure) 
it may be desirable to calculate the cost objective program evaluation twice: 
first using total costs when successful outcomes are plentiful and second, 
using the number of client/weeks in the program as successful outcomes. 
Figure Two illustrates a form used to compute the cost effectiveness of a 
program. 



Program Costs 

In computing the costs, it is desirable to do all programs at the same 
time. That is, that the distribution of administrative costs is made to all 
the programs at the same time. Not only is it easier to do it this way, but 
it also helps to fcvoid errors, since the amounts distributed to each program 
can be viewed at once. 



Program Staff 

The breakdown of the staff cost for the programs are done on the 
program staff form. See Figure Three below. The program staff should not 
include *wy administrative personnel higher than the position of Rehabilita- 
tion Dir ector. The salaries are computed on a monthly basis and then 
distributed by percentage to each of the programs. The actual distribution 
of time across various programs is somewhat subjective* but not entirely so. 
Generally, the best way to distribute the time is to have the staff keep 
track of their time spent in various activities for a couple of weeks. Once 
the percentage of the staff time has been allocated to each program, this 
percentage is multiplied by the monthly salary which is shown in the salary 
column under that particular program. 



ERLC 



19 



O 



EXPENSES 


Vocational Evaluation 


Training Program A 


Training Program B 




MONTH 


PI ILl 

CUM, 


MONTH 


CUM. 


MONTH 


CUM. 






STAFF 


4,098 


12,294 


5,172 


15,516 


4,785 


14,355 






FRINGE @ 15% 


615 


1,844 


776 


2,327 


718 


2,153 






ADMINISTRATION 


292 


876 


1,350 


4,050 


1,650 


5,750 






SUPPLIES 


23 


69 


250 


1.250 


350 


1,200 






DEPRECIATION 


250 


750 


1,250 


3,750 


1,410 


4,935 






TRAVEL 


150 


450 


500 


1,500 


120 


300 






CONSULTANTS 


130 


390 


210 


1,250 


150 


475 






UTILITIES 


671 


2,013 


4,500 


15,500 


5,275 


18,750 






TELEPHONE 


55 


165 


250 


850 


85 


350 






TOTAL EXPENSES 


6,282 


18,751 


14,758 


45,993 


14,543 


48,268 






CONTRIBUTED 
STAFF SUPPLIES 






500 


1,500 


650 


2,150 






TOTAL COST 


6,284 


18,751 


15,258 


47,493 


15,143 


50,418 






c||frcccc||i 
OUTCOMES 


8 


34 


5 


21 


5 


g 






COST PER OUTCOME 


786 


552 


2,460 


2,262 


3,039 


5,602 






GRANT SUPPORT 


4,713 


16,495 


10,250 


36,550 


725 


3,250 






GRANT 

COST PER OUTCOME 


589 


485 


1,708 


1,740 


145 


361 






SELF-SUPPORT 
COST PER OUTCOME 


197 


66 


751 


521 


2.894 


5.241 
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FIGURE THREE: Program Staff Worksheet 



STAFF NAME 


POSITION 


SALARY/MO 


VOC EVAL 


TRAINING A 


TRAINING B 




% 


SALARY 


% 


SALARY 


% 


SALARY 


% 


SALARY 




REHAB OIR 


1750 


30 


525 


40 


700 


20 


350 








SECRETARY 


750 


50 


375 


40 


300 


10 


75 








EVALUATOR 


1250 


100 


1250 
















CLERK 


700 


80 


500 


20 


140 













COUNSELOR 


1100 






inn 


nnn 












COUNSELOR 


950 










inn 


Q^n 

you 








PLACEMENT 


1100 


10 


110 


60 


660 


30 


330 








SUPER 
TEXTILES 


750 


10 


75 






90 


675 

Kit <J 








<?MPPP 
oUrtrt 

FURNITURE 


725 


on 


14b 


OU 


ODO 


oO 


217 








our tK 

MAINT 


800 


30 


240 


30 


C*t\J 


20 


isn 








SUPER 
CAFE 


775 


20 


155 


50 


387 


10 


78 








SUPER 
WAREHOUSE 


650 










90 


585 








SUPER 
UPHOLSTER/ 


775 


20 


155 


50 


387 


10 


78 








SUPER 
TX 


850 


10 


85 


30 


255 












SUPER 
DOCK 


725 


20 


145 


20 


145 


60 


435 








STORE 
MANAGER I 


750 


10 


75 


30 


225 


40 


300 








STORE 
MANAGER II 


690 










80 


552 






PROGRAM STAFF COST TOTALS 


4,098 


5,172 


4,785 
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Fringe benefits 



Fringe benefits are not only the cost of hospitalization, retirement 
plans, etc, paid by the agency, but also workmen's compensation costs, 
unemployment compensation and the employers share of social security. 
These costs are usually stated as a percentage of gross payroll. 



Administration 

Administration in this context relates to the cost of the executive 
director, his staff, the accounting department, personnel, and portions of 
other line staff time. The distribution of administrative costs can be done 
in many ways. See the Materials Development Center publication Determin- 
ing Effective Overhead Rates for Pricing Goods and Services for possible 
methods. 
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EXERCISE TWO: Describe the type of evaluation most suited to your 
facility needs. 
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SELF-TEST: Chapter One 



1. Evaluations have one thing in common, they answer 
about the operation of programs. 

2. Evaluation questions fall into one of five different areas: 



and f 

3. Three types of evaluations are common. They are: 
, and 

4. Three approaches are often used for evaluation purposes: 
s and 

5. Cost effectiveness a'iociates 

with . 
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SELF-TEST: Chapter Oae 
ANSWER KEY 



1. Evaluation have one thing in common, they answer questions about the 
operation of programs. 

2. Evaluation questions fall into one of five different areas: Needs as- 
sessment, operating descriptions, objective attainment, client flow, 
continuation determination. 

3. Three types of evaluations are common. They are: formative evalua- 
tions, summati ve evaluations, and needs assessment. 

4. Three approaches are often used for evaluation purposes: objective 
attainment, syste ms analysis, and cpst effectiveness. 

5. Cost effectiveness associates positive outcomes with the costs of ob- 
taining the outcomes. 
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Chapter Two 
Designing a Program Evaluation 



Planning for Program Evaluation 

Probably the biggest problem that program evaluators face is the fact 
that program objectives are often stated in ways that are difficult to 
measure. Regardless of the evaluation approach that a facility administrator 
decides to use, it is essential that the basic documents of program planning 
be used to develop assessment procedures. Program mission, goals, and 
objectives, as well as such administrative policies as admissions criteria and 
client flow paths must be examined and redefined in measurable ways. 

This redefinition is usually needed because the facility mission, goals, 
objectives, etc. arc determined through a political process involving many 
"stakeholders." 



The Political Process 

"Stakeholders" are individuals or organizational groups that have an 
interest in the outcome of a program evaluation. Because the type of 
evaluation desired, the approach chosen, the model selected, the questions 
asked, the objectives measured, the comparison criteria selected, the re- 
search design used, and virtually every other facet of the evaluation design 
will require judgments that will effect the results obtained by the evalua- 
tion, all stakeholders need to be provided with input into the planning and 
implementation of a program evaluation. If, in fact, they are denied a 
voice, they may discount, disbelieve, or ignore the results obtained. Thus, 
all individuals that will be affected by the evaluation should be included in 
the evaluation, beginning with the initial question formulation stage. 

These stakeholders should also be involved in the final reporting stage 
to ensure that the daf \ gathered wil! lead to actions taken. Because the 
end purpose of prograu* evaluation is to judge the program being evaluated, 
the "bottom line" for program evaluation is putting the results of the 
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evaluation to use. In order to be useful, the results must have basic 
utility. 



Seek Utility 

Program cvaluators must balance the agendas of many stakeholders 
when translating facility structural elements into measurable goals. If any 
of the stakeholders are ignored, the utility of the evaluation may be im- 
paired. Thus, to ensure the utility of program evaluations, an cvaluator 
should take under consideration the following facts: 

1. A balance must be struck among the differing value systems 
of stakeholders (facility administrators, funding agencies, and 
program clientele) by acting as a consultant to the develop- 
ment of the evaluation in the planning and formulation 
stages. 

2. Evaluations will probably be used to redistribute resources 
and, therefore, resistance to the evaluation is inevitable from 
some stakeholders. 

3. Program evaluation sometimes leads to decision making not in 
step with the successful achievement of program goals. 
Sometimes administrators make decisions that seem contrary 
to evaluation results for other, worthwhile, political reasons. 

4. Program evaluators best serve the stakeholders by focusing 
attention on the utility of the program evaluation goals. 



Structural Design Elements 

No matter which evaluation model is used, program evaluation must be 
linked to program planning and, thus, to the facility's mission, goals, and 
objectives. Facilities were established to meet real needs; over time, the 
mission-goals-objectivcs can become politically influenced, unclear, and 
difficult to measure. 



Inf luencer (Stakeholders) 

The mission, goals, and objectives of the facility arc influenced by 
interest groups. Other agencies, funding sources, and politically motivated 
client organizations attempt to influence your facility hoping to force you 
to meet their narrowly defined needs. These stakeholders will have a 
direct impact on your facility, either because they provide funding, referral, 
licensing, reporting, or regulatory pressures, or because they will change 
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your marketing patterns. The result of their influence will determine ihe 
mission and, in the long run, the programs offered by your facility. Ex- 
amples of such groups are: 

Consumers 

State vocational rehabilitation agencies 
Other purchasers of service 
J.T.P.A. & P.I.C. organizations 
United Way 
Public officials 

Association of Retarded Citizens 
City, county, and state boards and commissions 
Local and regional businesses and industry 
School districts 

Accrediting and licensing bodies 

Individual and foundation donors 

Granting agencies 

Wage & Hour compliance boards 

Etc. 



Mission Statement 

The most general statement of a facility's purpose is its mission 
statement. It spells out the purpose for the continued existence of its 
component programs. 

Generally, the mission statement is found in the facility's governing 
documents. In those situations where the governing document statements 
are not adequate to clearly differentiate and define the purpose of the 
facility a new mission statement should be developed, and approved by the 
governing body. 

Mission Statements Have Broad and Specific Characteristics 

The mission statement is probably the least difficult and one of the 
most important components of program evaluation. It should be written in 
such a way that it describes, in general terms, what the facility is trying 
to accomplish, the services provided, and who it serves. It should be broad 
enough to cover all of the programs offered by the facility included in the 
program evaluation system, but should not limit the scope of potential 
programs or possible client groups. Although broad and general, the mis- 
sion statement needs to be specific enough to distinguish one facility from 
others. The mission statement should contain the following elements: 
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1. Who is served 

2. Service provided 

3. Expected outcome 

4. Features that distinguish one facility from other facilities 

5. Specifics that enable goals and objectives to be set 

6. Geographic and other relevant dimensions 



Examples of Mission Statements 

Mission statements must be tailored to each individual facility. Below 
are some examples: 



Sample mission statement A 

To provide vocational evaluation, work and personal adjust- 
ment training, skills training and sheltered employment to physi- 
cally and mentally disabled and vocationally handicapped persons, 
which will help to decrease their social and vocational depen- 
dence and increase their ability to obtain an earned income. 



Sample mission statement B 

To assist persons who are physically and mentally disabled 
and otherwise vocationally handicapped to maximize their in- 
dependent living capabilities and enhance employability and 
economic independence through the provision of personal and 
social adjustment training programs, work adjustment, and shel- 
tered employment. 



Sample mission statement C 

To provide vocational rehabilitation and habilitation pro- 
grams such as work adjustment training, extended employment, 
and pre-vocational training as well as supportive and personal 
adjustment services on a needs basis for adult recipients of XYZ 
county who are developmentally disabled, physically disabled, 
mentally ill or chemically dependent for the purpose of facilitat- 
ing employability and independent functioning. 
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Admission Criteria 



Two sets of admission criteria should be established. Each set pro- 
vides control over those who are served consistent with mission statements, 
facility resources, and community needs. 

The first screening set is the general facility admission criteria. The 
second screening set is the admission criteria for entrance into each in- 
dividual program. If the facility provides only one program or service, then 
only one set of criteria need be established. 

General facility admission criteria sets basic standards and require- 
ments for admission to any and all programs in the facility. Some examples 
of criterion used to screen applicants are listed below: 

1. Unemployed 

2. Minimum age of sixteen 

3. Able to care for own personal needs 

4. Third party sponsorship 

5. County/city resident 

6. Able to provide own transportation 

7. Program consistent with needs of client 

8. Capable of independent living skills 

9. Conduct not dangerous to self or others 

10. Physical within last year 

1 1. Diagnosed disability 

12. Vocationally handicapped 

13. Ambulatory or mobile non-ambulatory 

14. Willing to participate voluntarily 

15. Able to earn minimum wage (facility) 

16. Desire to obtain employment 



Facility Programs 

Evaluations must also identify the separate programs in which special- 
ized services are provided to eligible clients. The differences between 
programs and services should be understood; this helps properly establish 
the facility's program structure for program evaluation purposes. 

Program services are a defined set of actions with the purpose of 
providing assistance to clients. A program, on the other hand, provides a 
variety of coordinated services with the purpose of achieving specific goals. 
These goals are aimed at the achievement of the facility's mission. 

An activity, for example job readiness training, might be either a 
program or a service depending on whether it is provided by the facility as 
a free standing program meeting the criteria listed below or as a service 
provided as part of another program such as work adjustment. Job readi- 



ness may be a separately defined set of activities with staff, a budget, and 
a physical location. In such a case, job readiness is probably a program. 
On the other hand, job readiness may be a set of activities that is provided 
to clients entered in the work adjustment program. Every client accepted 
for work adjustment would be eligible for the job readiness training. Thus, 
the activity is a service, not a program. 

Each program should be clearly identified as either a program or a 
service. Programs include the following elements: 



A program goal statement 

Specific admission criteria 

A description of the services provided 

A description of the clients it serves 

An identity separate from other programs 

Identified staff members 

A budget 



Program Goals 

Program goal statements are specific descriptions that clearly convey 
to individuals in and out of the facility the components and characteristics 
of a program. These characteristics should include: clients served, services 
provided, and results to be achieved. Most facilities have more than one 
program. However, if a facility has only one program and plans only to 
continue the present set of activities, the mission and goal statements may 
be identical. 

It is much more likely that your facility provides more than one 
program. Therefore you must develop goal statements for each program 
that is provided. 

Goal statements should be specific enough to determine each program's 
objectives and the measures that will be used to determine objective 
achievement. Below are some sample program goal statements: 

For Vocational Evaluation Programs 

1. To determine the specific assets and liabilities of physically 
and mentally disabled individuals, to develop appropriate 
vocational objectives with and for the client, to recommend 
the services which will best contribute to his or her voca- 
tional rehabilitation, and to provide information to the refer- 
ring agency which assists in determining feasibility for other 
rehabilitation services. 
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2. To assess the capabilities and limitations of the vocationally 
handicapped to determine the appropriateness of referral for 
education, training, or employment resources, and to assist 
selected individuals to obtain employment through vork 
adjustment, job placement, and follow-up services. 

3. To provide a variety of assessment techniques to vocationally 
handicapped clients which will assist the vocational rehabili- 
tation counselor and the client in determining further ap- 
propriate vocationally related services or job placement. 

4. To assess vocational skills and interests, independent living 
skills, and behavioral skills of the mentally, physically, and 
socially disabled in order to provide appropriate vocational 
recommendations for utilization by referral/funding agency 
and/or placement team. 

5. To provide information acquired through the assessment of 
employment capabilities, to disabled and multiply handicapped 
individuals and to the referring agency, which assists in 
determining the feasibility for vocational services, and iden- 
tifies specific services needed by the individual in order to 
achieve individually developed rehabilitation goals. 



For Work Adjustment Programs 

1. To provide structured and planned, closely supervised, reme- 
dial work experience, including behavior modification and job 
readiness training to physically and mentally disabled in- 
dividuals in order to assist them in obtaining employment. 

2. To enable the mentally, physically and socially disabled to 
improve their behavioral and work skills through behavioral 
management, vocational counseling, and production activity in 
order to maximize their earned income. 

3. To develop, modify, and support employment behaviors of 
mentally, physically, emotionally, and/or developmentally 
disabled persons at a level consistent with the demands of 
their individually developed rehabilitation goals leading to 
optimal earned income. 

4. To provide employability development, support, social ser- 
vices, and counseling to the physically, emotionally, mentally, 



socially, and economically handicapped in order to assist them 
in attaining competitive employment, and to reach their 
highest level of independent functioning. 

5. To provide employability development in work habits, at- 
titudes and behaviors, and supportive services to unemployed 
and the physically, emotionally, and mentally handicapped 
persons which will assist them to obtain an earned income. 



For Sheltered Employment Programs 

1. To assist vocationally disabled adults to achieve optimal 
earnings through extended employment opportunities in a 
sheltered environment with long range goals of eventual 
competitive employment. 

2. To provide sheltered long term employment and other suppor- 
tive services to persons whose handicapping condition is a 
continuing barrier to competitive placement in order to 
obtain an earned income. 

3. To provide an authentic industrial environment to mentally, 
physically, emotionally, and economically handicapped adults 
in order to assist them in maximizing their employment 
potential. 

4. To enable the mentally, physically, and socially disabled to 
maximize their earned income through sheltered employment 
or to gain competitive employment by providing remunerative 
work, vocational counseling, job placement activity and 
follow-up. 

5. To provide work experience for those physically, emotionally, 
and mentally handicapped or underemployed whose disabilities 
limit them from obtaining competitive employment, and to 
provide transitional employment for those who are endeavor- 
ing towards competitive placement. 



Program Admission Criteria 

Program admission criteria differ from facility admission criteria. 
They describe the specific requirements imposed by individual programs. 
Clients must meet program cnteria in addition to the facility admission 
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criteria. The establishment and enforcement of program criteria ensures 
that the program serves those for whom the services were intended. 
Below are some examples of program admission criteria: 



For Vocational Evaluation Programs 

1. Employability questionable or unknown 

2. Specific service need undetermined 

3. Does not require one-to-one supervision 

4. Identifiable vocational handicap 

5. Unemployed/underemployed or industrially injured 

6. Third party sponsorship 

7. Possess basic self-care skills 

8. Recent physical examination 

9. Recent psychological examination 



For Work Adjustment Programs 

1. Able to obtain either sheltered or competitive employment 

2. Third party sponsorship 

3. Recent vocational evaluation with goals and objectives 

4. Able to benefit from program 

5. Potential to earn facility minimum wage 

6. Unemployed/underemployed or industrially injured 



For Sheltered Employment Programs 

1. Sixteen (16) years of age or older 

2. Low potential for competitive employment 

3. Potential to earn facility minimum wage 

4. Able to benefit from sheltered employment 

5. Able to Urease independent skills. 

6. Produce at 50% of minimum wage. 



Services Provided 

Services are groups of activities provided to clients accepted into a 
program. The identification of a service involves the development of a 
process outline. Staff are assigned to carry out these processes. The 
service provided is related to the achievement of the program goal. It is 
not necessary for all clients to receive all services within a program. 
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A logical relationship should exist between clients, the services provid- 
ed, and the program goal. Services must lead to the accomplishment of the 
goal. 

Below is a listing of the kinds of services which may be provided in 
various programs: 

Vocational Evaluation 

1. Orientation 

2. Psychometric testing 

3. General Medical 

4. Follow-up 

5. Written Vocational Evaluation Report 

6. Physical Therapy Screening 

7. Career Exploration 

8. Developmental Assessment 

9. Functional Living Skills Assessment 

10. Employability Attitudes Evaluation 

11. Job Matching 

12. Work Sampling 

13. Situational Assessment 

14. Psychological Assessment 

15. Vocational Counseling 

16. Independent Skill Assessment 

17. Behavioral Assessment 

18. Physical Capacity Testing 

19. Job Site Analysis/Modification 

20. On-the-job Evaluation 



Work Adjustment 

1. Orientation and Intake 

2. Work habit and Attitude Training 

3. Vocational Counseling 

4. Personal Counseling 

5. Vocational Skill Training 

6. Off Site Evaluation 

7. Off Site Training 

8. Job Seeking Skills Training 

9. Follow-up 

10. Group Counseling 

11. Behavior Modification 

12. Role Modeling 

13. Grooming and Hygiene Skills 



36 

47 



14. Task Analysis 

15. Pre- Vocational Skills Training 

16. Personal and Social Development 

17. Paid Work 

18. Adult Basic Education/GED Preparation 

19. Psychological Consultation 

20. Controlled Work Experience 

21. Referral Services 

22. Social Case Work 

23. Activities of Daily Living Training 

24. Job Placement 



Sheltered Employment 

1. Vocational Counseling 

2. Referral Services 

3. Controlled Work Experiences 

4. Remunerative Work 

5. Job Development 

6. Work Skill Training 

7. Contract Work 

8. Leisure Time Activity Development 

9. Independent Living Skill Development 

10. Remedial Education 

11. Activity Therapy 

12. Personal and Social Adjustment 

13. Placement 

14. Job Seeking Skills 



Clients Served 

In addition to establishing admission criteria, disability types and other 
client characteristics of those who will be served should also be provided in 
written statements. A "client served" list should be developed. There 
should be a logical relationship between clients and the goal and services 
that will be provided. Some of the population groups you may serve are: 

1. Physically/Orthopedically disabled 

2. Emotionally disabled 

3. Spinal Cord injured 

4. Economically disadvantaged 

5. Hearing disabled 

6. Visually disabled 
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7. Neurologically disabled 

8. Developmentally disabled 

9. Learning disabled 

10. Culturally disabled 

11. Muitipiy disabled 

12. No previous work experience 

13. Severely disabled 

14. Substance abusers 

15. Public offenders 

16. Epileptics 

17. Non-English speaking 

18. Industrially injured 



Your list should include all types of disabilities and conditions, both 
primary and secondary, which could be served according to the na'ure of 
the program and the mission of the agency. Therefore, do not necessarily 
limit the list to those types of individuals already receiving service. On 
the other hand, the goals and nature of some programs will limit the dis- 
ability populations admitted. 



Program Objectives 

Objectives are often used as the statements tested by evaluation 
research. Evaluations need to discriminate among several different types of 
objectives. They include: process objectives, outcome objectives, and 
management objectives. 

Objectives are specific statements of the results a program intends to 
achieve. The objectives for each program should be consistent with the 
goal statement for that program. If all of the stated objectives are 
achieved, the program will have accomplished its goals. 

Objectives are the statements from which measures are derived, thus 
they must be stated in terms of the ultimate results or outcomes which 
clients of that program should achieve as a result of the services provided. 
Objectives must reflect both effectiveness and efficiency measures. Effec- 
tiveness measures tell evaluators how successful the client has been in 
achieving the benefits of the program. Efficiency measures tell evaluators 
how successful the program was at minimizing costs and/or time. Every 
program has at least one effectiveness and one efficiency objective. 
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Outcome Objectives 



Outcome objectives specify the status of the client following the 
provision of services* For example: employed, in training, etc. are specifi- 
cations of a client's status. They should reflect the benefits which clients 
may achieve as a result of the program. 



Process Objectives 

Process objectives are internal monitors of client progress. They are 
most often used to describe programs. For example: sheltered employment, 
extended employment, day activity centers, etc. are specifications of pro- 
gram activities in which clients may be spending many months or years 
before achieving a final outcome. Process objectives do not substitute for 
outcome measures. Process measures may reflect such things as: 

"Percent of clients who obtained a wage increase." 

"Percent of clients who move to a more independent living situa- 
tion," 

"Percent of clients who obtain an improved score on a social 
behavior checklist." 



Management Objectives 

Management objectives typically reflect the activities undertaken for 
"the good of the facility." Examples of such objectives are: "establish a 
vocational problems screening unit," "obtain additional funding," or "increase 
funding from present sources." 



Principles of Writing Objectives 

To help program evaluators create research models that provide solid 
data about the facility: 

1. Objectives should be measurable. 

2. Objectives should be achievable. 

3. If all of the objectives are accomplished, then the goal will 
have been achieved. 

4. The developer should keep in mind, the close relationship be- 
tween goals, objectives, and measures. 
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5. Objectives should be listed in rank order as to their impor 
tance with the most important objectives listed first. 



Examples of Program Objectives 



Vocational evaluation 

The primary objectives in evaluation should be consistent 
with obtaining positive results regarding the establishment of 
vocational objectives for the client. Other objectives are pos- 
sible, though of less relative importance. As with every program, 
efficiency measures also need to be included. Below are samples 
of vocational evaluation program objectives: 

Maximize client completions with positive vocational 
recommendations 

Maximize client enrollment for community service 
recommendations 

Maximize percent of terminees who complete their 
planned program 

Minimize the average program length of all program 
participants 

Maximize percent of clients on whom critical voca- 
tional recommendations are followed by counselor 

Maximize acceptance of primary recommendations by 
referral or funding agencies 



Work adjustment 

The primary objective in a work adjustment program should 
be directed toward achieving job placement for the client with 
competitive employment as its major emphasis. Also of impor- 
tance is the successful placement of the client in the vocational 
areas most suited to his skills and abilities as determined in 
evaluation as his vocational objectives. The time the client 
spends in his training should be minimized. Below are some 
examples of possible work adjustment program objectives: 

Maximize the percentage of clients who obtain full- 
time competitive employment 
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Maximize the percentage of program tcrminces of all 
types that obtain employment less than full-time 

Maximize the percentage of terminees who enter into 
skill training or OJT programs 

Maximize the percentage of clients who enter sheltered 
employment programs 

Minimize the time in program for persons meeting 
positive objectives 

Maximize the wages of clients that are placed into 
competitive employment 

Maximize the percentage of clients who complete their 
planned programming 

Minimize expenditures (on a per client basis) for 
achieving positive program outcomes 

Minimize the time spent in the program by all program 
participants 



Sheltered employment 

The ultimate goal for clients within a sheltered employment 
program is competitive employment. Although this may be of a 
long nature for many of these clients, it should remain as the 
emphasis and, therefore, the primary objectives should reflect 
this. Disabled clients who no longer realistically have the pos- 
sibility of obtaining this objective should be closed out as shel- 
tered and either hired as disabled employees or referred to more 
suitable programming. Some possible program objectives for 
sheltered employment programs are: 

Maximize percent of persons who obtain competitive 
employment, OJT, or non-sheltered work. 

Maximize percent of persons exiting the program who 
obtain skill training. 

Maximize the earnings of all sheltered employees. 
(Note that this is also a process objective.) 

Minimize time in program for persons obtaining com- 
petitive work, non-sheltered work, or OJT. 

Maximize the earnings of trainees who have compe- 
titive outcomes 
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Maximize referrals to alternative community services 
for nonsucccssful trainees. 



Setting Goals And System Structures 
Administration Vs. Program Structure 



Administrative Structure 

All facilities arc structured to implement the goals of the facility and 
produce results. This structure is usually displayed in the form of an 
organizational chart. 

Program Structure 

Program structure, in contrast, arranges the organization into distinct 
elements for the purpose of evaluation. This structure helps the facility 
know the degree to which it is achieving client benefits following the 
provision of services. It should be noted that the program structure for 
evaluation purposes is generally different from the organizational structure. 

Principles of Program Structure 

1. Program structures can vary between facilities, even those 
facilities with identical purposes. 

2. Program structure should be designed in such a way as to 
maximize the effectiveness and efficiency of management 
decisions. 

3. Program structure should be developed in such a way as to 
measure all clients entering the facility in at least one of 
the programs. 

4. Program structure will more than likely vary from the cur- 
rent organizational structure. 
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Creating Program Charts 



Program structure charts arc the starting point for program evaluation 
because they outline the mission and goals for the facility. Program evalu- 
ators determine how well the agency is achieving these goals. 

The easiest and most useful way to lay out the structure of a facility 
programs is to establish a flow chart as illustrated in Figure Four. The 
major structural elements arc shown on this chart. 

The chart also shows the organizations that influence the facility: its 
mission, the programs provided, and the specific goals, admission criteria, 
services provided, and the types of clients served for each program. This 
illustration shows the information in each box. For instance, the box 
labeled "mission statement" will contain the complete mission statement for 
the facility. 
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FIGURE FOUR: Sample Program Structure Chart 



ELK MOUND INTEGRAGED INDUSTRIES 
PROGRAM STRUCTURE CHART 



Influencer 



Influencer 



Influencer 



Mission Statement 



Admission Criteria 



Influencer 













Program A 




Program B 


Program C 












Program Goal 




Program Goal 


Program Goal 












Admission Criteria 




Admission Criteria 


Admission Criteria 












Services Provided 




Services Provided 


Services Provided 












Clients Served 




Cliants Served 


Clients Served 
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EXERCISE THREE: Develop a Program Structure Chart 



Using the charts on the following pages (or similar forms that better 
meet your facility's needs) create a program structure chart. 
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EXERCISE THREE: Continued 



PROGRAM STRUCTURE WORKSHEET 



LIST INFLUENCES: 

1. 

2. 

3. 

4. _ 

5. 

6. — 

7. 

8. 

LIST MISSION STATEMENT: 



LIST FACILITY ADMISSION CRITERIA: 

1. 

2. 

3. 

4. 

5. 

6. 

7. . 

8. 
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EXERCISE THREE: Continued 

PROGRAM STRUCTURE CHART 



INFLUENCER 



INFLUENCER 



INFLUENCER INFLUENCER 



1. 




2. 




3. 




4. 



















MISSION STATEMENT 
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EXERCISE THREE: Continued 

PROGRAM ELEMENTS WORKSHEET 

PROGRAM TITLE: 



PROGRAM GOAL: 



ADMISSION CRITERIA: 



1. 
2. 
3. 
4. 



SERVICES PROVIDED: 



1. 
2. 
3* 
4. 
5. 
6. 
7. 



CLIENTS SERVED: 



1. 
2. 
3. 
4. 
5. 
6. 

•7 . 
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PROGRAM OBJECTIVES: 



EXERCISE THREE: Continued 



PROGRAM ELEMENTS CHART 



PROGRAMS 



PROGRAM A 



PROGRAM GOAL 



ADMISSION CRITERIA 
I 



1. 
2. 
3. 
4. 



SERVICES PROVIDED 
I 



1. 
2. 
3. 
4. 



CLIENTS SERVED 



1. 
2. 
3* 
4. 



PROGRAM B 
I 



PROGRAM C 



PROGRAM GOAL 
I 



PROGRAM GOAL 
I 



ADMISSION CRITERIA ADMISSION CRITERIA 



1. 
2. 
3. 
4. 



SERVICES PROVIDED 
I 



1. 
2. 
3. 
4. 



CLIENTS SERVED 
1 



1. 
2. 
3. 
4. 



1. 
2. 
3. 
4. 



SERVICES PROVIDED 



1. 

2, 
3. 
4. 



CLIENTS SERVED 



1. 
2. 
3. 
4. 
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SELF-TEST: Chapter Two 



1. Program objectives are often stated in ways that are 



interest in the outcome of a program evaluation. 

3. Program evaluation data must have basic 

4. Program evaluation should be linked to _ 



are individuals or organizations that have an 



5. Evaluation planners should examine several facility policy areas before 
formulating an evaluation plan. They are: > 



and 



6. The mission statement is one of the 

and components of evaluation planning. 

7. Facility admission criteria set 



and for entry into all facility programs. 

8. A service is a defined set of provided within a program. 

9. A program has: , 

and . 



10. Goal statements should be specific enough to provide a 

of , how the are 

and the of 



11. insure that services are provided 

to clients who need and can benefit from their provision. 

12. Program services lead to the accomplishment of the" . 



13. are statements describing - the specific results to be 

achieved. 

14. Program structure charts provide evaluators with a of 

the facility's operation. 
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SELF-TEST: Chapter Two 
ANSWER KEY 



1. Program objectives are often stated in ways that are difficult to mea- 
sure. 

2. Stakeholders are individuals or organizations that have an interest in 
the outcome of a program evaluation. 

3. Program evaluation data must have basic utility. 

4. Program evaluation should be linked to program planning. 

5. Evaluation planners should examine several facility policy areas before 
formulating an evaluation plan. They are: Mission statement, facility 
admission criteria- program entities, program goals, program admission 
criteria, program services, client populations, program objectives, and 
program structure. 

6. The mission statement is one of the least difficult and most important 
components of evaluation planning. 

7. Facility admission criteria set basic standards and requirements for 
entry into all facility programs. 

8. A service is a defined set of actions provided within a program. 

9. A program has: goals, specific admission criteria, and several services. 

10. Goal statements should be specific enough to provide a determination of 
program outcomes, how the outcomes are measured, and the length of 
service provision. 

11. Program admission criteria insure that services are provided to clients 
who need and can benefit from their provision. 

12. Program services lead to the accomplishment of the program objectives . 

13. Objectives are statements describing the specific results to be achieved. 

14. Program structure charts provide evaluators with a map of the facility's 
operation. 
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Chapter Three 
Measures for Goals and Objectives 



Measures are statements that indicate how program goals and objec- 
tives will be achieved. Measures should reliably represent achievements. 
They are applied following the provision of services. Measures must clearly 
indicate who, what, where, and when they are to be applied. Some basic 
principles are: 

1. The measures should determine whether the objective was 
achieved (valid/reliable). 

2. The measures should report accurate information. 

3. The measures should indicate a final result occurring follow- 
ing the provision of services (except in the case of process 
objectives). 



Measures are best stated in terms of real numbers, averages, per- 
centages, time, or money. They should clearly reflect the method you will 
use to describe the provision of services that meet your objective. There 
should be no grey areas. The client either fits the criteria for measure- 
ment or does not fit the criteria for measurement. This gives stakeholders 
solid grounds for agreement on the actual amount of services that have 
been measured. Cook and Cooper (1978) state that measurement is the 
process of converting evaluation information into data that can be objec- 
tively interpreted. 

Measurement can occur on four different levels. Assumptions for each 
level allow successively more accurate analysis methods to be used to 
interpret evaluation findings. These measurement levels are: nominal scale, 
ordinal scale, interval scale, and ratio scale. 



Levels of Measurement 
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Nominal 



Nominal scales are the simplest. They are classifications (groupings) 
of people, things, or other units. Even when numbers are used to represent 
different classifications, relationships do not exist between the separate 
classifications. Thus, &o numerical analysis of the numbers can be made. 
For example, a nominal scale may be used to class people as 1) amputee, 2) 
paraplegic, 3) blind, 4) mentally retarded, 5) other. If all clients were 
assigned these nominal numbers, determining an average or any other 
statistic using the 1 - 5 scale would give no meaningful data. Additionally, 
nominal scales do not have zero points. 



Ordinal 

Ordinal scales improve the nominal scale. They can be used when 
some degree of relationship exists between units. Ordinal scales are often 
used to describe subjective data. For example, if we asked clients how 
satisfied they were with the training they received, we may classify their 
ansv/ers on a scale of 1 to 5 with very satisfied being one and very un- 
satisfied being five. However, the amount of satisfaction between 1 and 2 
is not necessarily the same as between 3 and 4. Thus, conclusions drawn 
from such data need to be carefully considered. Additionally, ordinal scales 
may or may not have a zero point. Ordinal scales are often used to evalu- 
ate rehabilitation programs. 



Interval 

Like ordinal scales, interval scales assume a direct relationship be- 
tween scale units. Unlike the ordinal scale, interval scale units are equally 
separated. The amount of change from 1 to 2 must be equal to the amount 
of change from 4 to 5 and for any other full unit within the scale. Addi- 
tionally, interval scales do not have a zero point. For example, IQ tests 
use interval scales and do not have zero points. To have zero IQ is to be 
unable to take the test (dead). Much of the objective data obtained for 
program evaluation can be placed on an interval scale. 




Ratio 

The highest level of measurement is the ratio scale. Like the interval 
scale, intervals between units have a direct and proportional relationship 
between themselves, but ratio scales always have a zero point. Thus, the 
data that is collected using a ratio scale must be capable of not existing. 
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Remember the Politics of Evaluations 



When determining program evaluation measures, it is essential to recall 
that program evaluation is a political process. All stakeholders have a vital 
interest in the measures that are chosen. Some stakeholders, in fact, will 
have legal rights to regulate the measures that are used. 

Where choices can be made, measures should be chosen to show the 
most programmatic impact relative to contemporary standards. For example, 
assume that it is possible to measure success by either the number of 
clients successfully entering competitive and/or supported work sites or by 
the total number of clients successfully leaving a training program. If we 
know that the principle source of program funds are obtained from a fund- 
ing agency that is particularly under pressure to place clients into sup- 
ported work programs, il will be politically expedient for the facility to 
choose that measure to examine when evaluating their program. 



Consider the Data Gathering and Analysis Techniques to be Used 

Another important area to consider when choosing measures for evalu- 
ation goals and objectives is the data gathering tactics and the analysis 
techniques that will be employed after data is gathered. Evaluators can 
employ the tactics of experimental research, correlational research, and case 
study to gather their evaluation data. And they can employ a wide variety 
of analysis techniques including multiple regression, chi-square, systems 
analysis, hypothesis testing, and sign tests to determine the meaning of the 
gathered data. All these tactics and techniques influence your data gather- 
ing techniques. It is essential that data gathered about the chosen measure 
be compatible with the technique that best answers the evaluation question. 
(We will examine these tactics in more detail in Chapters Four, Six, and 
Seven.) 



Some Possible Measures 

Figures Five, Six, and Seven provide a general comparison of possible 
measures for several types of facility programs. You will, of course, tailor 
the measures you use to the tactics of your program. 



55 

ERJC 66 



FIGURE FIVE: Possible Measures for Vocational Evaluation Programs 





(A) 


(B) 


(C) 


(D) 


(B) 


(F) 


ff 


OBJECTIVES 


MEASURE 


APPLIED 


TIME 


DATA 


GOAL 


1 


Maximize client com- 
pletions with posi- 
tive vocational 
recommendations 


% of 
clients 
with + 
voc ob- 
jective 


All 

closed 
clients 


At 
pro- 
gram 


Gen- 
erate 
by 

CVQJL 

uator 


85% 


2 


Minimize average pro- 
gram length for 
program completers 


Average 
program 
length 


All 

clients 
com- 
pleting 
program 


At 
pro- 
gram 
exit 


Gen- 
erate 
by 

aval- 
uator 


4 

w^eks 


3 


Maximize percentage 
of clients for whom 
critical vocational 
recommendations are 
followed by counselor 


% of 
clients 
with 
follow- 
up 


All 

clients 
com- 
pleting 
program 


60 

days 

after 

exit 


Gen- 
erate 
by 

coun- 
selor 


80% 




FIGURE SIX: Possible Measures for Work Adjustment Programs 





(A) 


(B) 


(C) 


(D) 


(E) 


(F) 


# 


OBJECTIVES 


MEASURE 


APPLIED 


TIME 


DATA 


GOAL 


1 


Maximize percentage of 
clients who obtain 
competitive 
employment 


% Of 

clients 
who 
obtain 
jobs 


All 

closed 
clients 


60 

days 

after 

exit 


Gen- 
erate 
by 
em- 
ploye: 


40% 

: 


2 


Maximize percentage of 
clients entering skill 
training 


% of 

clients 

who 

enter 

progrms 


All 

closed 
clients 


60 

days 

after 

exit 


Gen- 
erate 
by 
Ed. 
cnslr 


15% 


3 


Maximize percentage of 
clients obtaining 
sheltered employment 


% of 

clients 

who 

enter 

program 


All 

c? osed 
clients 


60 

days 

after 

exit 


Gen- 
erate 
by 

Prog. 
Mangr 


20% 


4 


Maximize wages for 
clients attaining 
objective number one 


Average 
wage at 
closure 


Clients 
reaching 
ob j ec- 
tive No. 
one 


60 

days 
after 
place- 
ment 


Gen- 
erate 
by 
Em- 
ploye: 


$3.75 


5 


Minimize time in 
program for clients 
attaining objectives 
one, two, and three 


Average 

length 

in 

program 


Clients 
reaching 
cb j ec- 
tives $ 
1, 2 & 3 


60 

days 

af^er 

exit 


Gen- 
erate 
by 

prgrm 
mangr 


24 
weeks 
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FIGURE SEVEN: Possible Measures for Sheltered Employment Programs 





(A) 


(B) 


(C) 


(D) 


(E) 


(F) 


# 


OBJECTIVES 


MEASURE 


APPLIED 


TIME 


DATA 


GOAL 


1 


Maximize percentage of 
clients who obtain 
competitive employment 


% Of 

clients 
who 
obtain 
jobs 


All 

closed 
clients 


60 

days 

after 

exit 


Gen- 
erate 
by 
Em- 
ploye: 


25 


2 


Maximize percentage of 
clients obtaining 
skill training 


% of 

clients 

who 

enter 

school 


All 

closed 
clients 


60 

days 

after 

exit 


Gen- 
erate 
by 

Ed* 
Cnslr 


15 


3 


Minimize time in pro- 
gram for clients 
attaining objectives 
number one and/or two 


Average 
length 
in the 
program 


Clients 
closed 
from 
ob j ec- 
ive 1/2 


60 

days 

after 

exit 


Gen- 
erate 
by 

prog, 
mangr 


52 


4 


Maximize earnings of 
sheltered employees 


Average 
wages 
per hr. 


All 

active 

program 

clients 


Every 
6 

months 


Gen- 
erate 
by 

book- 
keepr 


$2.50 


5 


Maximize percentage of 
clients utilizing 
program services 


% of 
clients 
who use 
program 
service 


All 

active 

program 

clients 


Every 
6 

months 


Gen- 
erate 
by 

prog, 
mangr 


75% 
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Determine Who/What Will Be Measured 



Your statement of who or what will be measured must indicate the 
groups of clients whose results will be calculated for that measure. 



Set Time of Measurement 

Time of measurement is the point in time when outcome information is 
collected. It is important that the time of measure be made in relationship 
to the program flow and the objectives to be measured. Some measures are 
taken at program exit (for example: time in program or identified vocational 
objectives) while others may have to be taken after an extended period (for 
example: obtaining competitive employment, or acceptance of recommenda- 
tions.) You must be sure that the time of measurement is clearly specified 
to ensure that the program impacts on the outcome. 



Specify Data Sources 

Specifics of where the information or data concerning the outcome of 
the objective will be located or stored should be recorded. Each client 
should have a "data sheet" in their main file that contains all the summary 
information on the client that will be used in the program evaluation. (The 
development and use of client data sheets will be discussed in Chapter 
Four.) Some sources of data are: 

Final evaluation reports 
Referral agency records 
Counselor records 
Client records 
Employers 
Vocational schools 
Program managers 
The clients themselves 
Bookkeepers 



Document Goal Expectancies 

Goal expectancy is the level of measurement that indicates an objec- 
tive has been achieved. Thus, it is the criteria against which actual per- 
formance is measured. Obtaining client performance data is the first step 
in determining program performance. If the facility is to find the data 
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useful, it should compare results against some explicit criterion or expec- 
tancy. 

If goals have not been specified, consistent judgments and subsequent 
action to improve program performance cannot occur. Initial goals may be 
established using the experience of your facility. The experience of other 
facilities whose programs are essentially the same can also be used. Fund- 
ing source expectations and information from literature related to the 
program being evaluated are other possible sources for expectancy data. 

A major benefit from setting goal expectancies is the process of goal 
setting itself. It forces staff to think about outcomes. Too often, rehabil- 
itation professionals can become so concerned with the process of rehabili- 
tation that they forget the need to obtain an outcome. The expectancy 
setting process may also improve communication among the staff. For 
example, a program director may believe that anything below 70% placement 
is unacceptable. The program staff, on the other hand, may feel that if 
they get 15% placed they are doing a great job. The administrator and the 
staff must discuss their differences in expectations and resolve them. 

Final expectations should be set by consensus between the facility 
director, the program director, and the program staff responsible for the 
delivery of the service. After data has been collected for a few months, 
goals may be reviewed and revised. If major changes are made in a pro- 
gram, goals should also be changed. However, once realistic goals are 
established, they should be maintained for at least one year. Establishing 
program evaluation goals at the beginning of the fiscal year may be a 
prudent step. 
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EXERCISE FOUR: Develop an Evaluation Measures Chart 



Using the chart below, or a similar form that better meets your 
facility's needs, create an evaluation measures chart for your facility. 



MEASURES FOR THE PROGRAM 





(A) 












# 


OBJECTIVES 


MEASURE 


APPLY 


TIME 


DATA 


GOAL 


1 














2 














3 














4 














5 
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SELF-TEST: Chapter Three 



1. Measures are statements that indicate how program goals and objectives 
be . 

2. Measures must be stated in of , 

3. The four levels of measurement are: , , , 

and P 

4. scales provide no numerical relationship among units. 

5. scalea provide a numerical relationship among units, 

but the relationships may not be equal. 

6. Equal numerical relationships among units are provided by , 

but they do not have zero points. 

7. The highest level of measurement is the scale. 

8. All have an interest in the measures that 

are chosen. ~^ 

9. Elements that should be considered when determining a measurement 

type for evaluation purposes are: or 

will be , the of the 3 the 

, and the . 
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SELF-TEST: Chapter Three 
ANSWER KEY 



1. Measures are statements that indicate how program goals and objectives 
will be achieved. 

2. Measures must be stated in terms of real numbers. 

3. The four levels of measurement are: nominal, ordinal, interval, and 

4. Nominal scales provide no numerical relationship among inits. 

5. Ordinal scales provide a numerical relationship among units, but the 
relationships may not be equal. 

6. Equal numerical reJationships among units are provided by interval 
scales, but they do not have zero poin* . 

7. The highest level of measurement is the ratio scale. 

8. All stakeholders have an interest in the measures that are chosen. 

9. Elements that should be considered when determining a measurement 
type for evaluation purposes are: who or what will be measured, the 
time of the measurement, the data source, and the goal expected . 
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Chapter Four 
The Collection of Data 



Evaluations of all types rely on the comparison of factual data to 
arrive at conclusions. They, therefore, can use the scientific method to 
assure that conclusions can stand public inspection. 

The scientific method is the systematic collection of factual data by 
objective observations to answer measurable questions. The difference 
between program evaluation research and traditional scientific research is 
that evaluation research must be used for decision making to justify expen- 
ditures of time and energy. Traditional scientific research may be per- 
formed for purely esoteric reasons. 



Research Tactics 

Evaluation researchers use three tactics to implement the scientific 
method. They are: experimental research, correlational research, and case 
studies. Brief descriptions of these tactics are provided below. A detailed 
discussion is found in Chapter Six. 



Experimental Research 

Experimental research is the gathering of factual data to prove or dis- 
prove a hypothesis. The hypothesis is a tentative solution to the evaluation 
question. For example, if a program is evaluated to determine its effec- 
tivrness at providing training to clients relative to another program, a 
hypothesis may be stated: Clients of Vocational Training Program A are 
more employable after completing the program than clients of Vocational 
Training Program B. 

To investigate this hypothesis using a scientific approach, it is neces- 
sary to choose the individuals that enter each program using a random 
sampling technique that allows all client referral to be equally likely to be 
entered into one or the other of the programs. This random assignment 
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helps reduce possible reasons for one group to be more employable to the 
program they hc*d entered. It is al',0 necessary to choose a measure (see 
Chapter Three) that represents employability. The measure must be quanti- 
fiable to allow data collection to proceed. In the language of researchers, 
the measurement is called the dependent variable because the data gathered 
will be dependent on the program from which it is gathered. Thus, each 
program is called an independent variable. 



Correlational Research 

Correlational researchers attempt to find associations or relationships 
between program components, other programs, and environmental considera- 
tions. This type of research does not provide any implication of cause. 
That is, a correlational study does not establish a hypothesis against which 
the data from dependent and independent variables are compared. 



Case Studies 

The case study is probably the most widely accepted data gathering 
tactic used by program evaluators in rehabilitation settings. In the case 
study tacric, existing data files on programmatic activities are examined to 
generate the data for use in making scientific comparisons. This tactic 
has many data gathering flaws that cause problems of interpretation for 
evaluators. The data are easily "contaminated" rendering any finding sus- 
pect. However, especial*^ when ongoing collection systems are in place, 
this approach may be the least costly of the research tactics. 



Ongoing Collection Systems 

Programs of a long duration, such as sheltered employment programs., 
may not generate sufficient client closures in outcome objectives to allow 
frequent program monitoring. Forms and procedures must, therefore, be 
developed to allow you to monitor all your clients in all programs based 
upon process objectives alone. This allows "long term* programs to be 
monitored on a quarterly, semi-annual and/or annual basis. Process objec- 
tives based on an ongoing data collection system supplement outcome mea- 
sures and allow for the interval monitoring of your program. 
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Set Procedures to Insure Accuracy and Completion 



For program evaluators to accurately interpret the data that is ob- 
tained using any of the above tactics, procedures must be developed to 
insure that the data that is collected includes all possible variables, and/or 
identifies single variables with precision. Two possible methods may be 
employed: the ongoing collection of global data and statistical sampling. 

Collection of all data points is not necessarily the most accurate 
method, though the proponents of ongoing collection of data often cite 
accuracy and completeness as a reason for the expenditure of time and 
effort. In fact, the global gathering of data for use in program evaluation 
may not only be a waste of data collection resources, it may actually be 
less accurate than representative sampling. This is true because more 
people are involved in global data collection, leading to a greater possibility 
for contaminated (inaccurately recorded) data. 



Statistical Sampling 

To insure that data samples are representative of the total data pool 
that is to be measured, samples must be randomly chosen to give each pool 
unit an equal chance of being chosen. 

Advantages 

Random samples that accurately reflect the population that they repre- 
sent, have three advantages over the collection of data from the population 
total: 

1. They provide real time and resource saving with little loss of 
accuracy because data is less likely to become contaminated. 

2. They can produce more timely results because the time of 
sampling can be quickly determined and the data quickly col- 
lected. 

3. They give program administrators more flexibility in the 
evaluation questions they ask because of their increased 
dollar and hour efficiency. 



Randomization Concepts 

Several descriptors are important to the creation of randomized sam- 
ples: 
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Population mean: The sum of all the units of a population 
divided by the number of units in the population. 



Sample standard deviation: A statistical measure of variability 
among sample scores. It is determined by subtracting the mean 
of the population from each score, squaring the products of the 
subtraction, summing the squared products, dividing the sum by 
the population total minus one, and finding the square root of 
the product of the division. (See "Estimating Mean Sample Sizes" 
below for an example using this formula.) 

The selection of a random sample to insure accurate representation of 
the population most commonly uses one of three methods: a table of 
random numbers, a computerized random sample, or a "seat-of-the-pants" 
randomization method. 



Tables of random numbers 

Tables of random numbers are found in many mathematics books. 
They are made up of a randomly generated series of numbers that can also 
be randomly accessed. An example of a random number table is found in 
Figure Eight below. 



FIGURE EIGHT: Table of Random Numbers 



76484 


83095 


41282 


48852 


51764 


23954 


15324 


81036 


62749 


33349 


67455 


55698 


70853 


00414 


73689 


94686 


69123 


85183 


78599 


85691 


29459 


20966 


53998 


19380 


60725 


10204 


69175 


30197 


86799 


39741 


40321 


63908 


24052 


88913 


70212 


81172 


07767 


97531 


07586 


97569 


14474 


93453 


26053 


25650 


81477 


19401 


70255 


36507 


58981 


47716 


23038 


27154 


80615 


88717 


33872 


41629 


57078 


85952 


61034 


26440 
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Using a random number table is one of the easiest methods of select- 
ing a random sample. All population units are first assigned a sequential 
number. Then, the evaluator chooses any point on the table to begin 
listing numbers. If the population contains less than 100 units, only two 
digit numbers are chosen; if less than 1000 units, only three digit numbers, 
etc. Any number that is greater than the population total is discarded. 
Numbers continue to be selected by reading down or across the table until 
the total needed for a representative sample has been reached. Then the 
randomly selected numbers are matched to the corresponding numbers 
assigned sequentially to each population unit. 



Computer generated samples 

Some statistical software programs are designed to produce random 
numbers. This may be the quickest way to match a randomly selected 
number to a population unit. Sequential numbers are assigned to population 
units. The computer selects the representative sample, and the randomly 
selected numbers are matched to the corresponding number assigned sequen- 
tially to each population unit. 



"Seat-of-the-pants" methods 

If only small samples are needed, it may be feasible to draw numbers 
out of a hat or some other equally folksy container to obtain a random 
sample. The trick is to insure that all units are equally represented and 
that each unit has the same chance of being chosen as part of the sample 
as all other units. 



Estimating Sample for Proportions and Means 

Another key to insuring that samples are representative is to accu- 
rately determine the number of units that must be included in the sample. 
Two processes for estimating the most accurate sample size are used. One 
is principally employed when the program evaluation question can be 
phrased such that the measurement obtained through the data gathering 
process will result in a population proportion figure. The other process is 
employed when the data gathering process results in a mean (average) 
figure. 
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Estimating proportional sample sizes 

m 

Three factors affect the determination of a sample size: the margin 
of error, the degree of risk, and the estimation of the true proportion. 

The degree of risk is a percentage representation of a reasonably 
accurate estimate of the true proportion. For example, we may want our 
final prediction to be accurate within 5% of the actual proportion for the 
data we are collecting. We would then specify a probability of .05. 

The margin of error is the probability that we will accept that the 
true proportion will differ from the sample by more than a small percent- 
age. For example, we may want to be sure that the actual proportion for 
the data we are collecting is within the margin of error at least 95% of the 
time. We would then specify a margin of error (represented in formulas as 
ME) of .05. 

The estimation of the true proportion is simply an educated guess at 
what the true proportion may be. 

We also need to determine a statistical amount that denotes a portion 
of a normal distribution curve. This amount is symbolized by the figure 
Za/2 . It is determined by using a table such as in Figure Nine below: 



FIGURE NINE: Table of Za/2 at Various Degrees of Risk 

a .01 .05 .10 

Za/2 = 2.57 1.96 1.65 

For example, assume that we wish to discover the proportion of cli- 
ents who have been discharged from our facility into competitive vocational 
settings in the last ten years. Let us further assume that our facility has 
served 1,500 clients during this period. Because we wish to use the results 
of this evaluation to show how effective our program has been compared to 
a similar facility in the southern part of our catchment area, we want to 
be fairly precise. Thus we set our degree of risk at .05 and our margin of 
error at .05. After talking with the vocational supervisors, we conclude 
that a reasonable estimate of the true proportion of clients who have been 
discharged into competitive vocational settings (this estimate is denoted as 
P A in our formulas) is about 34%. 

We nov^f use the following formula to determine the number of client 
files that we need to check to, with reasonable precision, determine the 
actual percentage: 
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(Za/2) 2 P (1-P) 

(Sample n = 

Size) 

ME 



Inserting our predetermined figures into the formula, we find that our 
sample size w n M equals 345 cases. 



(1.96) 2 (,34) (,66) 

(Sample n = 

size) 

(.05) 2 



Because this equation was designed to determine the size of samples 
when they are very large relative to the population, a smaller sample may 
be used when the product of the division of the sample size (n) by the 
population size (N) is more than .05. 

Dividing 345 by 1500 we determine the product of .23. Because this is 
larger than .05 we can use the following formula to obtain the actual, 
smaller, sample size: 



(Sample size) 
n 

(Smaller n' « 



sample 1 + (n/N) 

size) 



Filling the equation with our predetermined figures we note that: 



345 

(Smaller n 1 » 



sample 1 + .23 

size) 

Therefore, to be 95% certain that the true proportion falls within 5% 
of the figure determined by the sample we will need to select a random 
sample of 280 cases from the 1,500 cases the facility has on record. 
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Estimating mean sample sizes 



Some program evaluation questions are better answered using mean 
(average) measures rather than proportional measurements. In this case, in 
addition to the margin of error and the degree of risk that we are willing 
to accept (sec the discussion "Estimating proportional sample sizes" above) 
we must also determine an estimate of population variance. Population 
variance is the square of the standard deviation, thus we would solve the 
equation for standard deviations up to the point where the square root is 
determined to obtain the population variance. 

Rather than guessing at the population variance, which could lead to 
extremely inaccurate estimates of sample size, it is best to conduct a simple 
pilot study. By randomly choosing cases to examine using the measurements 
that have been chosen, we can obtain an estimate of the population vari- 
ance to use in the formula. 

For example, assume that we wish to discover the average amount of 
time that a client spends in our facility before being placed into a compet- 
itive vocational setting. In the last ten years, our facility has served 1,500 
clients. Because we wish to use the results of this evaluation to show how 
effective our program nd$ been, compared to a similar facility in the south- 
ern part of our catchment area, we want to be fairly precise. Thus we set 
our degree of risk at two months and our margin of error at .05. We have 
decided to pull a sample of 50 cases as a pilot study to determine an 
estimate of population variance and also decided to define our measurement 
as months from program admission to discharge into a competitive setting. 

After randomly selecting 50 cases from all the clients who entered our 
facility in the past ten years, we discover that only 15 cases resulted in a 
placement in competitive vocational settings. Because we decided to use 
months in programming as our measurement, we list each case with the 
length of tenure as bciow: 



Case 




Case 


Months 


1 




8 


8 


2 




9 


10 


3 


2 


10 


16 


4 


1 


11 


5 


5 


5 


12 


17 


6 


7 


13 


4 


7 


14 


14 


15 






15 


7 



To estimate the population variance of the pilot study we subtract the 
mean of the population from each score, square the products of the sub- 
traction, sum the squared products, and divide the sum by the population 
total minus one. The formula for these steps is: 
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_ 2 
E (Y - Y) 
Sample Variance = 



n - 1 



(E is a mathematical symbol that means "the sum of.") 



Continuing our table we determine the population mean, subtract the 

mean from each score, and square the product of the subnotion: 

Y _ _ 2 

Case Months Y - Y (Y - Y^ 

1 20 11.2 125.44 

2 1 -7.8 60.84 

3 2 -6.8 46.24 

4 1 -7.8 60.84 

5 5 -3.8 14.44 

6 7 -1.8 3.24 

7 14 5.2 27.04 

8 8 - .8 .64 

9 10 1.2 1.44 

10 16 7.2 51.84 

11 5 -3.8 14.44 

12 17 8.2 67.24 

13 4 -4.8 23.04 

14 15 6.2 38.44 

15 7 -1.8 3.24 



E - 132 E = 538.40 

Y = 132/15 = 8.8 

Now we can plug the data into our formula: 



538.4 

Population variance = = 38.46 

14 
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At this point we have nearly all of the data points needed to estimate 
the sample size: degree of risk of two months, .05 margin of error (.95 
confidence level), and 38.46 population variance estimate. One more point 
needs to be determined. That is the Za/2 normal curve computation. 
We obtained z Za/ 2 figure of 1.96 from Figure Nine above. 

Now we are ready to use the formula for estimating the sample size. 

It is: 



(Za/2) 2 sd 2 



(Sample n = 

size) 0 

ME 



Entering our predetermined data, we calculate that our random sample 
should be drawn so that it includes at least 37 valid cases. 



(1.96) 38.46 



147.75 



n = 



= 36.9 



Because we drew 15 cases for our pilot test, we need to obtain a 
sample of only 22 more cases. Because the first fifteen were drawn from a 
sample of 50 cases (approximately 30%) we would need to draw a sample of 
73 cases (22 divided by .3) to be assured that 22 usable cases will be 
selected. 

As in determining sample sizes based on proportions, this equation was 
designed to determine the size of samples when they are very large relative 
to the population, a smaller sample may be used when the product of the 
division cf the sample size (n) by the population size (N) is more than .05. 

Dividing 37 by 1500 we determine the product to be .02. Because this 
is smaller than .05 we cannot use the formula for obtaining a smaller 
sample size. (The formula and its application, should your product be 
larger than .05, is found above in the section titled "Estimating Proportional 
Sample Sizes.") 
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Gather Data on Summary Sheets 



All information necessary to track any clients in facility programs may 
be gathered on a client data sheet (see Figure Ten). As the client enters 
the program, a client data sheet is started by the person in charge of that 
particular program. Fo* example, the vocational evaluator would fill out 
the form for those clients entering the evaluation program. This data sheet 
is placed in an "active client" file. All the data sheets for active clients 
are thus in a single folder. The "active clients" file remains in the pro- 
gram providing services at that time. 

The data collection sheet is divided into several major sections: 
General information, Benefit status, Recommendations, Follow-up/Referral 
Information, and Client descriptors. This format is modified by your 
facility to meet your particular data collection ne is. 



General Information 

This section contains information about the client, program status 
information, and client vocational objectives. This section is completed for 
every client at the time of each program entry. The dates the client exits 
and closes the program are also entered here. When the client exits a 
program and enters another, another client data sheet must be created. 

The exit time is the day on which the client completes a specific 
program. Closure is the day on which a case or program manager deter- 
mines that the client received (or did not receive but was terminated) the 
expected benefit from the program. Closure dates are determined for all 
objectives of each program when you measure each area. 

If the client's program is interrupted, the starting time for the inter- 
ruption is stated at the time it begins. The client data sheet is then 
placed in the "interrupted programs" file. When the client returns, that 
date is recorded on the form and it is returned to the "active clients" file. 

If a client terminates the program prior to completion, this informa- 
tion is entered on the form as well, along with the nature of the termina- 
tion (self or facr' y initiated.) (You may want to determine the number of 
self-initiated dropouts as compared to the number of agency initiated ter- 
minations.) 

When a client exits the program, the staff providing services, such as 
the vocational evaiuator, completes sections A and C on the clJent data 
sheet and sends it to the rehabilitation director or other personnel respon- 
sible for follow-up and program evaluation. 
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FIGURE TEN: Client Data Sheet 



A: GENERAL INFORMATION 



NAME- 



JOHN RICHARD BOTTERBUSCH 



ssi 



330-15-2345 



AGE- 



26 



-SEX-! 



ADDRESS 



14 EVALUATION PR, HEN0H0NIE, HI 5*751 



REFERRAL SOURCE- 



HI PVR 



-COUNSELOR- 



T. JOHNSON 



DirADILITIEC DIAGNOSED HEMAPLEGIA (338) 



-PH£ 



254-6825 



PROGRAM MANAGER: 

PROGRAM STATUS: ACTIVE 
011586 



RONALD FRY 



INTERRUPTED 
-TO 



PROGRAM T3RMINATI0N: SELF-INITIATED- 



EXIT 
031586 



CLOSE** 
031586 



SPAN 
8 WKS 



AGENCY INITIATED- 



VOCATIONAL OBJECTIVES (OR EVALUATION PLANS) : 

DOT COD E 203-262-018 Terminal System Operator 

205-367-042 Registration Clerk 



DOT CODE- 
DOT CODE- 



215-362-014 



Dispatcher Clerk 



B; BENEFIT STATUS 

OUTCOME OBJECTIVES: 

J . Maximum completions with positive recommen d 

2 



OBJ#- 
OBJ#- 



OBJ# 

PROCESS OBJECTIVES: 

QBJ £_1 . Evaluate on vocational objectives above 

obj if : 



NON-BENEFIT ACHIEVER- 



EXIT- 



Minimum mean prog length for nuncompletors 



Max. % of clients with followed recommends 



-EXIT- 



-EXIT- 



-EXIT- 



-CLOSED- 



EXIT- CLOSED- 



-CLOSED^ 



-CLOSED- 



-CLOSED- 



C : RECOMMENDATIONS 
VOCATIONAL OBJECTIVES: Evaluation indicates thai Registr Clerk feasible 
VOCATIONAL PLAN* Re ^ er *° Area One Tech Institute for instructional trng 



EMPLOYER/AGENCY- 



D: FOLLOW-UP/REFERRAL INFORMATION 
WI Div of Motor Veh «^ Tm *^ m R. Franke 



-CONTACT- 



ipH 5 ^232-6543 



JOB TITL E MVD A PP 11cat1ons Processor 

STARTING WAGE $ ^= 6 MONTH OTJ- 



-DOT CODE- 



205-367-042 



-12 MONTH OTJ- 



t 

c 
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E: CLIENT DESCRIPTORS 



PHYSICAL DISABILITY 

MENTAL ILLNESS 

MENTAL RETARDATION 

HEARING DISABILITY 

VISUAL DISABILITY 

DEVELOPMENTAL DISABILITY 
LEARNING DISABILITY 

EPILEPTIC 

MINORITY 

SUBSTANCE ABUSER 

PUBLIC OFFENDER 

MULTIPLE DISABILITY 

SEVERE DISABILITY 

INDUSTRIALLY INJURED ... 

FORMER CLIENT 

PUBLIC ASSISTANCE 

LESS THAN H.S. EDUCATION 
H.S. EDUCATION OR G.E.D. 
POST H.S. EDUCATION 
NO PREVIOUS EMPLOYMENT . 

UNDER 21 

MALE 

FEMALE 

PREVIOUS INSTITUTION . . . 

VR REFERRAL 

BLIND AGENCY REFERRAL . . 

SCHOOL REFERRAL 

JTPA 

WORKERS COMPENSATION ... 

SELF-REFERRAL 

DMHMR REFERRAL 

OTHER REFERRAL SOURCE . . 
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Next, the sheet is filed in an "exited client" file, flie closure date 
used for the measurement of any client should be the most distant date of 
closure for his or her objectives. For example, if the client exits from 
vocational evaluation, his client data sheet would be placed in the "exited 
client" file 60 days after exit, rather than for the current month, if any of 
his objectives are measured 60 days past exit. 



Benefit Status 

This section relates directly to the client's achievement of the objec- 
tives of the program in which he/she is enrolled. When closure occurs, it is 
determined if the client has or has not attained the program benefits. 
When this is determined, the client is closed in that program and his pro- 
gram evaluation sheet is placed in a "closed client" file. At the end of 
each evaluation period program evaluation sheets from this file will be used 
to compute program evaluation data. 



Recommendations 

This section, includes the client's established vocational objective and 
vocational plan recommendations. 

Follow-up/Referral Information 

This section is used to enter employer or other service agency infor- 
mation to be used in follow-up and referral. 



Client Descriptors 

Finally, the client descriptor section is located on the back of the 
client data sheet or on a separate sheet. The client's descriptors are 
checked on entry and allow th? facility's case population to be more nar- 
rowly and completely defined. 



Tabulate The Data 

Each program your facility includes in its ongoing data collection 
system will need to have its own results summary sheet to tabulate the data 
from closed client cases. The use of a summary sheet greatly simplifies the 
task and minimizes the amount of staff time needed. It allows for the 
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handling of each client data sheet only once. Each program summary sheet 
has space to record the results of 12 closed cases. Should you have more 
than 12 clients in closure status per program in any one month, simply use 
the number of sheets required and tally them on completion (see Figure 
Eleven). 



Summary Procedures 

Each month (or other regularly established period) the staff person(s) 
responsible for tabulating case data will take each program's "closed client" 
file and complete the summary sheets as follows: 

A. All information on the summary sheet is filled out one client 
ct a time. This allows you to handle each client data sheet 
only once. 

B. Staff responsible for tabulating the results will take each 
program's "closed client file" for the evaluation period and 
complete the summary sheet. 

The results of the facility's program objectives oy client descriptors 
gives the facility a feeling for how successful certain clients and groups of 
clients are in its programs. This may give you important information 
regarding how the facility is meeting the needs of these clients, which 
clients it dees well with and which it does not, which clients may be in 
need of a different type of program or different approach, etc. Information 
such as this can be useful in determining the direction of a facility's 
programs, new programs and services the facility may need to consider 
developing, and where to allocate its resources. 

C. The results for each program are now totaled in the last 
column (see Figure Twelve) for each parameter that you h&ve 
listed data for. 

D. The results for each program objective is now totaled for the 
current and cumulative periods. 

Program results by client descriptors are all in terms of the number of 
clients closed in each descriptive category. This is a tally of the number 
of clients closed achieving the objective listed divided by the total number 
of clients closed for the program. 
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FIGURE ELEVEN: Program Evaluation Results Summary Sheet 



TIME PERIOD: 


X 


2 


3 


A 

4 


o 


D 


•7 


Q 
O 


y 




1 OR 2 OBJECTIVES 


1 

1 


l 
i 


l 
i 






1 
1 




I 
X 


1 


*t/ to 


3 OR MORE OBJECTIVES 


1 


1 






- 


1 
1 




1 


1 

1 


H\)to 


NONBENEFIT ACHIEVER 








1 






1 






13% 


PROGRAM LENGTH (WKS) 


2 


4 


2 






2 




6 


6 


4.3 


JOB PLACEMENT 


1 


1 


1 




1 




1 


1 




40% 


WORK ADJUSTMENT 


1 




1 










1 


1 


27% 


SHELTERED EMPLOYMENT 




1 
















7% 


WORK ACTIVITIES 




















7% 


SKILL TRAINING 


1 










1 






1 


20% 


HOME BOUND 




















0 


OUT REFERRAL 










1 




1 






13% 


AGENCY TERMINATION 














1 






7% 


SELF-TERMINATION 








1 












7% 


PHYSICAL DISABILITY 


1 




1 








1 






AAA/ 

20% 


MENTAL ILLNESS 




1 




1 












13% 


MENTAL RETARDATION 




1 


1 




1 


1 








27% 


HEARING DISABILITY 
















1 




/% 


VISUAL DISABILITY 




















0 


DEVELOPMENTAL DIS. 




1 


1 












1 


one/ 

20% 


T ^% % VTT\T^ *^ --TP « m --TP V --TP m * * 

LEARNING DISABILITY 








1 












•70/ 

7% 


EPILEPTIC 


















1 


7% 


MINORITY 


1 






1 




1 


1 




1 


33% 


SUBSTANCE AuUSER 


1 


















7% 


PUBLIC OFFENDER 








1 










1 


13% 


MULTIPLE DISABILITIES 


1 


1 


1 


1 










1 


33% 
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FIGURE TWELVE: Program Results by Client Descriptors 
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Data Tabulation 



After the results summary sheets have been completed for each pro- 
gram, the staff responsible for collection and tabulation will complete the 
program results by objectives forms (see Figure Thirteen.) 

FIGURE THIRTEEN: Program Results by Objective 
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Frequency Distributions 



The conversion of raw scores from summary sheets and other data 
gathering forms into group centered scores is a standardization process that 
allows data to be more easily compared with other groups. Several stan- 
dardization methods will result in the expression of data in the form of a 
frequency distribution. They include: percentile ranking, histograms, and 
f requ. ncy polygons. Frequency distributions are representations of data 
based on their relative numerical groupings. 



Percentile Ranking 

One of the most common ways used to represent individual scores in 
relation to groups is to indicate the percentile ranking of the score. 
Percentile ranks are sometime converted into cumulative rankings to indi- 
cate the actual location of a score within a large mass of data points. 
Figure Fourteen below illustrates how raw data (in this case, the number of 
weeks from program admission until successful discharge) is listed by fre- 
quency, percentage of the total number of individuals entered into the 
program, and cumulative percentile ranking. 

From this table, using both the percentage ranking and the cumulative 
percentages we can learn a great deal about the training program that each 
individual was engaged in. For example, we can see that the largest per- 
centage of clients (21.053%) complete their training in 15 weeks by the 
percentage figure for clients completing 15 weeks of training. More sig- 
nificant perhaps, is the finding that 52.632% of the clients complete their 
training in less than 17 weeks. Of even more interest is the fact that only 
1.754% of the clients complete training in less than 11 weeks, and that over 
70% of the clients complete training in from 11 to 20 weeks. 



FIGURE FOURTEEN: Weeks in Program 
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Histograms 

Histograms arc similar to percentile rankings because they also rank 
data from lowest to highest. However, histograms visually place the data 
into five to fifteen "ranges" allowing multiple individual scores to be more 
easily assessed. Each data range must be equal to all other data ranges. 
The size of each range is determined by the number of ranges and the size 
of the population. For example, the following table depicts the number of 
days of evaluation received by 100 clients who entered a training program. 
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Davs of Evaluation 



Frequency 



10-24 
25-39 
40-54 
55-69 
70-84 
85-99 
.'00-114 
115-129 
130-144 
145-159 
160-174 



2 
5 
15 
20 
25 
i5 
10 
4 
3 
0 



Building a histogram from the table, we create a visual representation 
of the data that will allow us to clearly see the variance within the pro- 
gram. 
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Computing Standard Scores 



Another method of raw score standardization is the computation of a 
standard score. Scleral methods are used. Like the process used for 
frequency distribution, one method is the percentile rank which we have 
already discussed. Especially useful in the development of standard scores, 
using a percentile ranking method is the use of the cumulative percentiles. 
Standard scores using the above methods convert each raw score into a 
score that can be related to other scores in the same group. 

Another method for determining a standard score is to compute a 
derived score. A derived score differs from a group centered standard 
score in that it provides standardization that allows comparisons between 
studies. Two easily computed standard scores arc in common use: the Z 
score and the T score. 



Z Scores 

The Z score transformation converts a raw score into a score that 
reflects the raw scored relationship io the group norm. This relationship 
is expressed as a figure that indicates how many standard deviations the 
raw score is from the group's mean. Thus, deviation from the mean can be 
compared for scores from any group. 

Because deviation from the mean can be positive or negative, Z scores 
can also be positive or negative. In addition, it is rare to find score 
deviations that greatly exceed three standard deviations from the mean, 
therefore, Z scores usually range from -3.0 to 3.0 with 0.00 indicating the 
mean. 

Z scores are computed by determining the group mean, subtracting the 
raw score from the mean, and dividing the product by the group standard 
deviation. Thus the formula for computing a Z score is: 



Z = X - X 



SD 



Let's use the example above regarding the time that a client spend, in 
our facility before being placed into competitive vocational settings. 

You may recall that fifteen cases in the pKot study resulted in a 
placement in competitive vocational settings: 
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Case 


Months 
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To determine the group's standard deviation we must find the square 
root of the population variance. This is determined by subtracting the 
mean of the population from each score, squaring the products of the 
subtraction, summing the squared products, dividing the sum by the popula- 
tion total minus one, and finding the square root of the product. The 
formula is: 



/ _ 2 

/ E (Y - Y) 

Standard Deviation = / 



V n - 1 



Our tabulations would look like Figure Fifteen below: 
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FIGURE FIFTEEN: Sample tabulation of standard deviation 
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Now we can determine the Z score for any of the raw 
scores. For example, the Z score for case 9 is determined by converting 
the raw score, 10, using the following process: 



Z = X - X Z = 10 - 8.8 = 1.2 = .19 



SD 6.2 6.2 
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T Scores 



Based on the Z score, the process for determining a T score is de- 
signed to eliminate both decimals and negative numbers. 

After a Z score has been determined, it is multiplied by 10. To the 
resulting product is then added 50. This creates a standard score distribu- 
tion from about 20 to about 80 with 50 as the mean. 

For example, using the Z score for case 9 in the above sampling of 
months to successful closure we would use the following formula to deter- 
mine a T score: 



T = (Z X 10) + 50 T = (.19 X 10) + 50 T = 51.9 
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EXERCISE FIVE: Determining Sample Size for a Proportional Study 



Determine the size of sample that you would choose using the following 
data and problem. You must choose a margin of error and a probability of 
risk. Defend your choices. 



The Director of Rehabilitation Services, Inc. was concerned that the 
percentage of clients referred to the placement program from janitorial 
crews was lower than the 'vera#e in the wood products work station. This 
was a concern, because the janitorial crew program required a higher 
productivity rating for admission than the wood products area and should, 
therefore be providing quicker remediation. In addition, the county Depart- 
ment of Social Services was charging that the crew supervisor was holding 
clients in the program longer than necessary to insure that crew work was 
completed. Thus, either the training that was provided in the wood prod- 
ucts area was superior to the crew or the measurements used by the crew 
supervisor were inaccurate. Other explanations for the difference (if a 
difference actually existed) included: inappropriate application of produc- 
tivity measures by crew staff, inappropriate application of productivity 
measures by wQod products staff, subjective application by either area's 
staff, failure of the janitorial supervisors to move clients out of the pro- 
gram after task mastery, etc. 

The first task confronting the Director was to determine the per- 
centage of clients referred to the placement program by the janitorial crew 
supervisors. The Director had determined this percentage for the wood 
products area in an evaluation completed last year. Because the comparison 
data included wood products clientele up through last year, the Director 
must search the files of clients who were entered in the janitorial crew 
program through last year. 

The crew supervisor provides the Director with a list of all clients 
that had entered the program in the 13 year old program. It contains 372 
names. The supervisor also told the Director that experience (the super- 
visor had been with the facility for five years) indicated that about 33% of 
the clients who entered the training program were eventually referred to 
the placement program. 

How many files should the Director sample to estimate the true per- 
centage? 
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1. My margin of error will be 

2. My risk of error will be 

3. My Za/2 based on the risk is 
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EXERCISE SIX: Worksheet for Sampling Means 



Determine the size of sample that you would choose using the following 
data and problem. You must choose a margin of error and a probability of 
risk. Defend your choices. 



The Director of Rehabilitation Services, Inc. was concerned that the 
average productivity of clients on the janitorial crews after one month of 
training was lower than the average in the wood products work station. 
This was a concern, because the janitorial crew program required a higher 
productivity rating for admission than the wood products area. Thus, either 
the training that was provided in the wood products area was superior to 
the crew or the measurements used by the crew supervisors were inac- 
curate. Other explanations for the difference (if a difference actually 
existed) included: inappropriate application of productivity measures by crew 
staff, inappropriate application of productivity measures by wood products 
staff, subjective application by either area's staff, failure of wood products 
supervisors to move clients out of the program after task mastery, etc. 

The first task, in any case, is to determine the average productivity 
of the janitorial crew trainees after they had received one month of train- 
ing. The Director had determined this average for the wood products area 
in an evaluation completed last year. Because the comparison data includes 
wood products clientele up through last year, the Director must search the 
files of clients who spent at least one month training on janitorial crews 
up through last year. 

The crew supervisor provides the Director with a list of all clients 
that have received at least one month of training in the 13 year old pro- 
gram. It contains 278 names. The supervisor also told the Director that 
experience (the supervisor had been with the facility for five years) indi- 
cated that the average percentage after one month was likely to be 68% 
and that variance was likely to be from 60% to 75%. 

How many files should the Director sample to estimate the true aver- 
age? 

1. My margin of error will be . 

2. My risk of error will be . 

3. My Za/2 based on the risk is . 

4. My estimate of population variance is . 
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EXERCISE SEVEN: Create Data Gathering Forms 



Create forms that will allow the staff of your facility to summarize the 
following data in the most efficient manner. 



1. Percent of clients competitively employed 

2. Percent of clients entering each program 

3. Percent of clients receiving supplemental income benefits 

4. Percent of clients receiving placement training 

5. Average hourly earnings of clients in each program 

6. Average client productivity percentage in each program 

7. Average working hours 

8. Average length of training in each program 

9. Percentage of clients self-terminating their training 
10. Any other measure that your think" will be valuable* 
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SELF-TEST: Chapter Four 



1. Evaluations rely on the of data to expected data 

to provide information on which to base decisions. 

2. The scientific method is the collection of 

by to answer 

questions. 

3. Evaluation researchers principally use three evaluation tactics. They 

are: , , 

and _ . 



4. allow facility administrators to ask 

questions about the operation of their facility both after services have 
been provided and while they are being provided. 

5. Data samples are chosen by methods to insure that the samples 

are representative of the total data pool. 

6. Random samples insure accurate data collection while providing 

and . 



7. Three factors affect the determination of a sample size: the 

of , the of , and the of _ 

or from the , 



8. Ongoing data collection systems gather client statistics on a 



9. allows data to be more easily compared between groups. 

10. are representations of data based on 

relative numerical groupings. 

11. Percentile ranks provide an intergroup comparison of . 

12. Two common standard scores are the and the . 

13. The reflects the raw score's relationship to the group on 

the basis of standard deviations from the mean. 

14. are based on and eliminate decimals and nega- 
tive numbers. 
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SELF-TEST: Chapter Four 
ANSWER KEY 



1. Evaluations rely on the comparison of factual data to expected data to 
provide information on which to base decisions. 

2. The scientific method is the systematic collection of factual data by 
objective observations to answer measurable questions. 

3. Evaluation researchers principally use three evaluation tactics. They 
are: experimental research, correlational research, and case studies. 

4. Ongoing data collection systems allow facility administrators to ask 
questions about the operation of their facility both after services have 
been provided and while they are being provided. 

5. Data samples are chosen by random methods to insure that the samples 
are representative of the total data pool. 

6. Random samples insure accurate data collection while providing time 
and cost savings. 

7. Three factors affect the determination of a sample size: the margin of 
error, the degree of risk, and the estimate of true proportion or vari- 
ance from the mean . 

8. Ongoing data collection systems gather client statistics on a client data 
sheet. 

9. Standardization allows data to be more easily compared between groups. 

10. Freouencv distributions are representations of data based on relative 
numerical groupings. 

11. Percentile ranks provide an intergroup comparison of raw scores . 

12. Two common standard scores are the Z Score and the T Score . 

13. The Z Score reflects the raw score's relationship to the group on the 
basis of standard deviations from the mean. 

14. T Scores are based on Z Scores and eliminate decimals and negative 
numbers. 
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Chapter Five 
Systems Analysis 



A system is an interactive relationship that exists between units, with 
the activity of one unit affecting the behavior of all other units. Programs 
within facilities are constantly transferring clients in and out to complete 
various phases of their rehabilitation program plans. Thus the facility is in 
a constant state of flux as internal and external relationships change. The 
daily status of the facility depends upon the interactions of each component 
rehabilitation progran through client flow, staffing, and budgetary decisions. 

The use of systems analysis for program evaluation is an attempt to 
describe the success of a facility relative to the flow of clients through the 
facility's programs. A complete discussion of systems analysis is found in 
Systems Analysis in Rehabilitation Facilities. (Smith, 1986.) 



Types of Systems 

Two types of systems can operate within an organization. They are 
closed systems and open systems. Of the two, only the open system is of 
major use for the analysis of facility operations. 

Closed systems have no regular input or output. A fire alarm circuit 
operates as a closed system. Once the system's circuit is closed, the alarm 
rings. Because the closed system has no other inputs or outputs, the alarm 
will continue to ring as long as the circuit is closed, even when no fire 
exists. A closed system is rarely found operating in facilities. 

In an open system, inputs and outputs can vary in intensity and 
duration. For example, a pot of water on a stove can be examined as a 
system. Heat is applied and absorbed by the pot until the water reaches its 
boiling point. When the water begins to boil, the heat input is no longer 
used to raise the temperature of the water, it is used to convert the water 
to steam. Thus a change has occurred in the system's output. If the 
amount of heat input is constant, the steam output will also be constant. 

The system formed by the pot of water and the stove eventually 
reaches a state where the heat and steam outputs are constant. This state, 
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the state at which inputs and outputs do not fluctuate, is called a "steady 
state." When a rehabilitation program is begun, it will take time before the 
first clients complete the training process. At the beginning, clients arc 
cither filling programs or leaving through self-termination (dropout) or 
discharge. Eventually, the number of clients completing the training pro- 
gram, plus all terminations will equal the program's client intake, and the 
program system will have achieved a "steady state." For example, if an 
evaluation program is designed to be completed in one month, a training 
program four months, and a placement program one month, the minimum 
time required for the facility system to reach a steady stetc condition is 
six months. 



Dynamics of Systems 

Facility systems principally exhibit two dynamics: feedback loops and 
feedback delays. Both dynamics arc important when analyzing facility 
operations. 



Feedback Loops 

Virtually every system requires feedback from its component units. 
Systems also need mechanisms (or regulators) that respond to the feedback. 
If no regulator is present or the regulator is not sufficiently sensitive to 
process feedback data, many problems may develop. An example of a 
feedback mechanism is a thermostat in a heating system. A thermometer on 
the thermostat registers the temperature in the house. The thermostat is 
programmed to trigger the furnace control when the temperature (as regis- 
tered on the thermostat) drops below a preset level thereby providing 
"feedback" to the furnace, acting as a feedback mechanism. Turning on the 
furnace raises the level of heat in the house. As the temperature rises, 
the thermometer on the thermostat also rises and, when the temperature 
reaches a predetermined desired temperature, triggers a switch that turns 
off the furnace, again providing feedback. 

Program evaluation provides feedback to rehabilitation programs like a 
thermometer provides feedback to a heating system. Without program 
evaluation there is no way to determine if the program is producing the 
desired results. Programs without evaluation feedback continuously provide 
the same training curriculum or make occasional, arbitrary changes. 
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Feedback Delay 

Time is required for feedback data to be transmitted to and confirmed 
by the action generating parts of the system. This time is called "feedback 
delay/ 1 For example, within the system of "higher education" the prestige 
of a university is gauged by the accomplishments of its graduates. Because 
at least five years must pass following graduation for most graduates to be 
noted as majcr contributors to their chosen fields, the present reputation of 
the university Is really based on the actions of the university five years 
earlier. Rehabilitation programs arc also judged by "cid" data, though the 
"old" data has becu delayed only about two to five months. 

Some facility decisions need to be made before feedback data is avail- 
able. For example, a student niay choose to enter a university degree 
program because existing data shows promising future openings for gradu- 
ates of that curriculum line. Unfortunately, that data on employment, even 
if not subject to collection delays, will not be relevant for the job market 
four years later when the student graduates. 



System Processes 

Systems analysts look for the presence and dynamics of several pro- 
cesses that may, or may not, be at work in a facility system. They arc: 
1) Oscillation, 2) Routing, 3) Reservoir effects, 4) Trajectory, and 5) 
Valves. 



Oscillation 

"Oscillation" is the pattern of variation in system input and output. 
Input and output volumes fluctuate, Sometimes they arc relatively large, 
and at other times they ~rc relatively small. If these variations (oscilla- 
tions) decrease with time, the oscillation pattern for the system is called 
"dampened." 

Oscillations occur for five reasons: 1) Seasonal variations, 2) Feed- 
back delays, 3) Insensitive control devices, 4) Long response wmcs, 5) 
Ovcrrcactions to feedback information 



Seasonal Variations 

Some rehabilitation programs arc subject to seasonal variations in their 
client loads. Variations in funding periods, staff and client vacation sched- 
ules, and seasonal changes in weather conditions can cause client count 
oscillations. The client referral source or funding agency may also have 
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seasonal factors (such as fiscal periods) that will affect the program's client 
count. 

Feedback Delays 

Oscillations can also be caused by feedback delay. For example, 
training programs that prepare clients for high demand occupations will 
experience high demand until the need for workers in these occupations is 
met. However, the demand for training in these occupations is likely to 
continue even after the need for workers has been filled. Thus more 
workers will be trained in the area than are actually needed by the field. 
As trained workers fail to obtain jobs, it will soon become known that an 
overabundance exists for workers in the area. The demand for training 
will, then decline rapidly. The oscillation cycle will continue as lessened 
demand for training creates a trained worker shortage, etc. ad. infinitum. 



Insensitive Control Devices 

A third reason oscillation occurs is poor sensitivity of control devices. 
For instance, our bodies are far less sensitive to temperature changes than 
are thermostats. People who stoke wood furnaces know that the tempera- 
tures within their homes vary widely. This is true because the furnace is 
stoked when their owners perceive the need for more heat; when they stoke 
their furnaces, they put in enough wood to lengthen the time until the next 
stoking, thus insuring that the temperature rises. Compared with tempera- 
ture variations in homes equipped with thermostatically controlled furnaces, 
the wood heated home has a great range of temperatures. 

Insensitive control systems are also responsible for wide variations 
within facilities. Sensitively controlled rehabilitation programs usually have 
targeted referral rates. When referrals fall below this targeted level, the 
program administrators begin to actively solicit referrals. Waiting until the 
client referral rate is low enough to cause concern before attempting to 
increase the rate is certain to increase the range of client flow oscillations 
within the facility. 



Long Response Times 

System oscillations often accompany long response times. In a heating 
system, for example, the thermostat must be set to activate the furnace 
before the room temperature actually falls to a desired degree setting. This 
allows the furnace to respond to a new heat demand even as the tempera- 
ture in the house continues to fall. The thermostat also should be set to 
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trigger the furnace to stop producing heat before the optimum temperature 
has been reached, allowing the heat remaining in the furnace to deliver its 
final heat energy without overheating the house. 

In training programs with exit criteria established as attainment of a 
certain level of quantity or quality of production, a response time gap 
exists between the time that the criteria level is reached and the time that 
it is recognized. Thus, as a client narrows the gap between the exit cri- 
teria and current skill levels, the frequency of measurement usually in- 
creases. 



Overreactioc to Feedback Information 

The final reason that oscillation occurs is the tendency to overreact 
to feedback information. For example, when taking a shower it is natural 
to turn on both hot ?.nd cold faucets. Because the water from the "hot" 
faucet has been in the pipes for a while it comes out cold. To quickly 
discharge this f cold water, it is natural to turn the "hot" faucet to full 
volume. When the hot water finally arrives, it is scalding, prompting the 
bather to ^quickly reduce the flow. Often, the flow is slowed too much and 
must be adjusted again. After several oscillations between extremely hot 
and extremely cold a balance is struck. Responding to feedback oscillation 
in this manner is called "control by successive approximations." 



Routing 

In a systems context, routing refers to the establishment of review 
points. When a review point is reached, the program administrator decides 
which one of several routes the client, data, or materials will follow. 
Review points, routes, and routing rules may be in the form of documented 
policies or based on experiential judgments. 

Decisions tend to develop consistent patterns over time. An examina- 
tion of routing decisions made at the end of an evaluation program, will 
probably show that consistent percentages of clients are referred to the 
various work adjustment and placement programs available in the facility 
and community. These routing percentages are one of the most important 
tools of systems analysis. The analysis of system behaviors depends on the 
analysis of routing patterns as well as intake and discharge rules. 



Reservoir Effect 

Reservoirs perform essentially the same function in human service 
delivery systems as in water delivery systems: they dampen input oscilla- 
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tions. They are called "waiting lists" when an attempt is made to quantify 
the units in the reservoir by listing- each person eligible to enter the 
program. 

Water reservoirs lose holding units through evaporation. Human 
service delivery system reservoirs (waiting lists) also lose "units" over time. 
Potential clients remove themselves from the reservoir as their time on the 
list is extended. They are lost to other training opportunities and to 
disinterest. 



Trajectory 

The trajectory of a system is the apparent direction of movement for 
the system's units. According to Newton's first law of dynamics, a body in 
motion continues to move in the same direction with the same speed, unless 
an outside force acts upon it. The facility's system also tends to demon- 
strate this dynamic principle. When a program is initiated, the clients 
receiving training, evaluation, or placement services could be seen to be 
"put in motion" within the system. They are likely to stay in motion, in 
the direction projected, unless an effort is made to change client flow 
direction. 

The perceived trajectory as observed by persons that interact with the 
facility: the genera! public, potential customers, potential clients, and 
potential referral sources should always be considered when using systems 
analysis for program evaluation. This "image" of the program must be 
considered whenever programmatic changes are made. 



Valves 

A valve controls unit flow through a system. In facility systems, 
valves regulate the movement of clients. They may be located anywhere 
within the system. 

Some valves restrict the numbers of clients. For example, a program 
intake valve might restrict the number of clients that are admitted to the 
program through the use of policy statements. Policy statements could 
restrict client capacity by simply stating how many clients can be served at 
any one time. 

Policies regarding program entrance also act as values, restricting 
numbers by creating a screening device. For example, a pre-entrance 
staffing to screen appropriate clients is a system valve device because the 
staffing committee modifies the flow of clients into the program. 
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Visual Representations of Systems 



Systems analysis involves the definition, depiction, and examination of 
the relationships between units. Thus the methods of systems analysis 
involve the defining of system flow, the diagramming of system flow, and 
the manipulation of the flow using system models. To be useful to program 
evaluators, systems analysis data needs to be represented visually. The 
representational method used is determined by the data available, the re- 
sources to be expended, and the results desired. 



Representing System Flow 

System flow is the movement of people or things through the or- 
ganization following defined routes. Flow can be described numerically 
using actual unit counts, percentages of total units, or by trajectory. For 
example, movement of clients from evaluation into a vocational program can 
be stated as "Four clients per month," "Twenty-two percent of all referrals 
out of evaluation," or "Four referrals, vocational route." The method used 
is determined by the type of relationship that exists between system units, 
and the administrator's need for data. 

Often, numerical or trajectory definitions can be better conveyed 
through the use of a chart or graph. Visual representations of the quanti- 
tative and directional data will assist administrators and analysts in spotting 
areas of concern. 

Four types of visual representations are regularly used to analyze 
systems. They are: activity charts, layout charts, personal relationship 
charts, and data charts. 



Activity Charts 

An activity chart depicts the flow of people and things through pro- 
cessing centers. In fact, activity charts are sometimes called process 
charts. They are often used to analyze production operations. Each major 
part of the chart indicates an area where units are changed in some man- 
ner. Activity charts can be used to compare systems when similar symbols 
are used to denote activities at each system unit. 



Layout Charts 

Layout charts illustrate the actual physical locations of processing 
units within a facility area and the flow that occurs between the units. 
They differ from activity charts only in the fact that they represent actual 
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physical locations rather than abstract "unit" concepts. Actually, layout 
charts also incorporate elements of the activity chart because system flow 
is illustrated. 



Personal Relationship Charts 

These charts illustrate the relationships that exist between the deci- 
sion making parts of the facility system. They may depict the facility's 
organizational structure, or indicate who is responsible for making routing 
decisions. The flow lines that are depicted usually represent lines of 
authority and indicate the persons responsible for carrying out system 
defined tasks. 



Data Charts 

Data charts give visual representation to numerical information. Data 
charts can take many forms. They may be bar graphs, line drawings of 
many types, or simply graphic displays of data. Pie graphs are not nor- 
mally used to depict systems data because flow data is not easy to repre- 
sent using this form. The important elements of data chafing for flow 
analysis purposes are: equality of representation units and the use of time 
as an independent variable. 



Flow Analysis 

Flow analysis is a relatively simple process that allows administratqrs 
to create and examine system models, gathering data useful in preparing for 
program evaluations. 

The data needed to complete a systems flow analysis includes the 
following: 

1. Intake rates 

2. Routing rules ;ind rates 

3. Exit ports and rates 

4. Program tenure rates 

5. Capacities 

6. Waiting ust figures 



Intake Rates 



Intake rates should be reported both as the number of clients per 
program and the number of clients per point of entry. The figures are 
usually reported as average weekly or monthly figures. If oscillation of the 
intake figures is great, notations of the oscillation ranges will help the 
analysts round out their picture of the system. 



Routing Rules and Rates 

Routing rules and rates should also be described for the program 
evaluator. The possible pathways that clients may take in their travel 
through the program should be illustrated, including documentation of rules 
that dictate movement through various routes. 



Exit Ports and Rates 

Exit ports can be described as the means by which clients leave the 
system. Many ports may be described, such as: job placemeit, sheltered 
employment, self-termination, referrals, dismissal for cause, etc. The actual 
number of clients using an exit port and a percentage breakdown should 
also be determined. When examining exit data, the analyst will look for 
percentages that are constant over time or oscillations that cannot be 
explained by changes in client population or in the program composition. 



Program Tenure Rates 

Program length may be defined as a specific time period or an average 
of indefinite time periods. For example, some evaluation programs have set 
program lengths such as a standard one week program. All clients are 
given standard, time limited tests and complete the evaluation in exactly 
one week. On the other hand, some evaluation programs start with a one 
day assessment that is standard to all clients, but the length of the rest of 
the evaluation is determined by the tests assigned after the initial assess- 
ment. In the case of time specific programs, the program length is self- 
evident. However, in non-specific programs, especially work adjustment and 
skill training programs, the varying length of time that clients spend in the 
program requires that an average be established. 
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Capacities 

Program capacity may be set by many factors such as: desirable staff 
to client ratios, space, equipment, or available work. In most cases, any of 
these factors can be changed, thus influencing program capacity. One 
reason that administrators turn to systems analysis for data is to obtain 
information that will allow them to make predictions of future program size 
and needs. System flow analysis provides data that will give the adminis- 
trator time to plan. 



Waiting List Figures 

The numbers of clients, or the lack of a waiting list, is valuable data 
for the systems analyst. 



Examining the Flow 

After gathering system data and building a system model based on the 
data, the data and the model are analyzed to gather information helpful for 
determining administrative courses of action, thus improving system func- 
tioning. 



FIGURE SIXTEEN: Simple Flow Path of an Evaluation Program 



Refer 




To determine the number of no shows, multiply the no-show rate (25%) 
by the number of referrals. Thus, ten referrals times .25 equals 2.5 clients 
per week who do not show. The number of referrals minus the no-shows 
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equals the number of clients who start evaluation each week. In this case 
it is 7.5. The dropouts from the evaluation program must be computed 
before the number of clients in a program can be estimated. In this ex- 
ample, 10% drop out per week. Thus, multiplying the number of clients 
entering the program (7.5) with the dropout rate (10%) results in the figure 
.75. Subtracting .75 (the dropout average) from 7.5 (the average number of 
clicks actually entering the program each week) results in 6.75 (the number 
of clients actually flowing through the entire four week program. 

To estimate the average number of clients in the program at any one 
time, we must determine the average point in the program that dropouts 
occur. In this example we will estimate that the average dropout occurs 
after two weeks of programming, or at the 50% point in the program. We 
must also take into account the four weeks it will take to bring the pro- 
gram to full capacity when it is initiated. 

If dropout occurs at the end of the first two weeks, we can assume 
that 7.5 clients per week occupy the program for the first two weeks, and 
6.75 clients per week occupy the second two weeks. Multiplying the num- 
bers for each week we determine a grand total of clients per week. Two 
times 7.5 equals 15 and 2 times 6.75 equal 13.5. Fifteen plus 13.5 equals 
28.5 or the number of clients in the program at any one time after steady 
state has been achieved. 

Staffing decisions made at the end of the program are illustrated in 
Figure Seventeen below: 



FIGURE SEVENTEEN: Sample Flow Path 



.67 / 10% 




In our example above of persons exiting an evaluation program, 50% 
are referred to training program A, ten percent directly to placement, 25% 
to other services, and 15% are not recommended for further services. 

To compute actual figures from the percentages, each percentage in 
the route is multiplied by the number of persons exiting the evaluation 
program. In other words, in this example 6.25 multiplied by .5 equals 3.38, 
the number of clients per week entering the training program. The num- 
bers ia Figure Seventeen show the clients achieving each outcome or pro- 
ceeding into other parts of the rehabilitation system. The next illustration 
(Figure Eighteen below) models the training program. 



FIGURE EIGHTEEN: Sample Flow Patfc 
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An average of 3.38 clients per week enter the training program from 
the evaluation program. Approximately one person per week also enters the 
training program by another input port. Thus, total client movement into 
the training program is 4.38 clients per week. Multiplying 4.38 by a pro- 
jected dropout rate of thirty percent we determine that an average of 1.3 
clients will dropout per week. Subtracting 1.3 from 4.38, we determine that 
3.1 clients will complete the program per week. Because the training 
program has an estimated length of 18 weeks and because we assume that 
dropouts occur half-way through the program, we can determine the average 
number of clients in the program during any week by multiplying the first 
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nine weeks by the total number of clients entering and the second nine 
weeks by the number entering less the dropouts. Thus, 4.38 multiplied by 9 
results in the figure 39.42, and 3.1 multiplied by 9 results in the figure 
27.1. Adding 39.42 to 27.9 results in the figure 67.32 which is the average 
number of clients that would be in the training program at any time after a 
steady state is achieved. 

We now need to examine the final system component, the placement 
program. A model of this program is illustrated in Figure Nineteen below. 



FIGURE NINETEEN: Sample FSow Path of a Placement Program 
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Because .67 clients per week enter the placement program through the 
evaluation program port, 3.1 clients per week from the training program 
port, and one client every two weeks, or .5 per week who enter from other 
ports, 4.27 clients enter the placement program each week. Because the 
placement program has a five week average length the number of clients in 
the program each week is determined by multiplying 4.27 with 5 to obtain a 
figure of 21.35. In this example, 25% percent of the placement program 
clients do not obtain jobs. Multiplying 4.27 with .25 results in an average 
weekly termination of LI. The program figures indicate that 65% of the 
clients do obtain a job in competitive sites. Multiplying 4.27 with .65 
results in 2.8 clients per week obtaining competitive positions. The remain- 
ing 10% (.427) of the clients are placed into sheltered employment positions. 
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Uses of Flow Analysis Information 



Flow analysis is useful for program evaluation in four primary ways. 
First, the analysis data, usually presented in activity or numerical chart 
forms, provides a "picture" of the facility's present operating status. 
Second, the "picture" of the present system can be used to predict changes 
that need to be made to keep the facility healthy (effective.) Third, the 
system "picture" will help the administrator "balance" system units to keep 
them operating with peak efficiency. Finally, the "picture" can be altered 
by the administrator to assess the feasibility of changes, particularly the 
introduction of new services. 



Create a "Picture 11 of the Present System 

After computing weekly figures for the whole system, a yearly sum- 
mary (which may be a prediction, or may be compiled from past activities) 
can be made. From these annual figures it is possible to predict program 
evaluation results. For instance, 105 persons would be placed in jobs from 
the training program. This was estimated by multiplying 3.1 clients per 
week who leave the training program by 65 percent, which is the placement 
rate, times 52. Because 105 people are placed out of the 228 persons 
served, a success rate of forty-six percent is determined for this program. 
In the placement program, 145 placements are made from a pool of 222 
persons entering the program resulting in a success rate of sixty-five 
percent. 



FIGURE TWENTY: Flow Summary Chart 





Evaluation 


Training A 


Placement 


Total 


Referrals 


520 


52 


26 


598 


Served 


390 


224 


222 


468 


Self -terminate 


39 


68 


0 


107 


No benefits 


92 


68 


7 


167 


Placements 


23 


105 


145 


145 



110 

1.19 



Predict the Effect of or Need for Changes 



Once a real or proposed rehabilitation program has had flow analysis 
performed, it can be used to predict the effect of changes in the system. 
For example, an analyst may observe that changes in referrais will cause 
the dropout rate in vocational evaluation to increase from ten percent to 
forty percent. By changing that figure and redoing the significant calcula- 
tions, the analyst may find that the annual number of placements will drop 
from 146 to 94. Additionally, the change in referral rate will effect the 
number of persons in work adjustment at any one time, dropping from 67.32 
to 49.77. While, this type of prediction does not solve referral problems, it 
does help administrators prepare for a problem's effects. Also, a little 
manipulation of figures shows that if the dropout rate is increased to forty 
percent in evaluation, the numbers in work adjustment and placement can 
be maintained - if the referrals to evaluation can be increased from ten to 
fifteen per week. However, this will also mean that the average number of 
clients in vocational evaluation will increase from 28.5 to 36. Other options 
are available, such as increasing the number of referrals directly into the 
training program, which will help the program return to it's needed load 
level. 



Balance System Units 

Possibly the most important use of flow analysis is to balance an 
existing program. For instance, if the capacity of the training program 
modeled above is 100, then this program is only operating at sixty-seven 
percent utilization. By a simple manipulation of the figures it becomes 
apparent that there will have to be 6.5 clients per week entering the 
training program in order to achieve the full load of 100 clients. If only 
one client per week continues to be referred directly to the training pro- 
gram, 5.5 clients per week must be referred from the evaluation program. 
An average of 16.2 referrals per week must, therefore, be made to the 
evaluation program to balance the system. If it is not feasible to increase 
referrals by six clients per week, perhaps the administrator should decrease 
capacity for the training program, allocating staff and space resources to 
other programs. Systems analysis may be used for all rehabilitation pro- 
grams. The effects of changes in client flow for any one program tends to 
affect all programs in the system. 



Demonstrate the Feasibility of New Services 

Systems analysis can be used to determine the feasibility of initiating 
new programs. For example, if a significant number of clients are 'jeing 
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routed to other services, the facility administrator may wish to determine 
the feasibility of starting an "in facility" program to address those needs. 
In the example above, a total of 2.7 clients (1.02 "no service" plus 1.68 
"other") fall in that category. If fifty percent of this group have a com- 
mon need then there is the potential of 1.35 clients per week that could 
enter the new program. If the new training program lasts 12 weeks, need- 
ing twenty-five clients to be cost effective, the program will never achieve 
a steady state equal to the cost of maintaining the program. See Figure 
Twenty-one below: 



FIGURE TWENTY-ONE: Illustration of a Projected Program. 
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Projections for proposed programs arc performed exactly as for exist- 
ing programs; estimated data is used when actual data is lacking. For 
example, when 1.35 (the projected intake figure) is multiplied by .2 (the 
projected dropout rate), the resulting figure .27 is an estimate of the 
average weekly number of dropouts. Subtracting .27 (the estimated dropout 
figure) from 1.35 (the projected intake figure) we establish a projection of 
the number of clients that will complete the program each week. Because 
the dropouts are expected to occur half way through the 12 week program, 
multiplying 1.35 by 6 results in an average first six week load of 8.1 and 
multiplying 1.08 by 6 results in an average second six week load of 6.48. 
Adding the figures results in a projected maximum load of 14.58 clients. 

For the proposed program to be cost effective, the administrator would 
need to solicit referrals from other sources and/or increase the number of 
clients referred from the evaluation program. The administrator must also 
consider the effect that any change in referral pattern may have on the 
facilities other programs. 
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EXERCISE EIGHT: Create a Systems Flow Chart 



Using the following data, construct a system flow chart that depicts 
the operation of Rehabilitation Services, Inc. 

Rehabilitation Services, Inc. provided services to 152 clients *n 19XX. 
Thirty-one clients received evaluation services, 134 received vocational 
training, and 18 received placement services. Of the 134 clients vho 
received vocational training, 26 received training in the pillow-making work 
station, 43 received training in the janitorial services program, 67 received 
training in metals salvage, and 39 received training in the wood shop. 
Some received training in more than one work station. Nineteen clients 
who received training in janitorial services had first received training in 
wood shop (8) and pillow making (11). Twenty-two clients in the wood 
shop had also received training in pillow-making (2), metals salvage (16), 
and janitorial services (4). Nine of the clients who received placement 
services were referred from the janitorial program. Five of the others 
were referred from evaluation. The remaining clients in the placement 
programs were direct referrals. Evaluation services referred 8 clients to 
each vocational program. Twelve clients received competitive jobs through 
placement services. The number of clients who self-terminated from each 
program wa^ exactly equal to tnc number of clients who were unsatisfac- 
torily discharged from each program. At the time the chart was drawn, no 
clients were being evaluated, 14 clients were in the pillow-making work 
station, 23 in the janitorial program, 28 in the wood shop, and 4 were in 
placement. 
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SELF-TEST: Chapter Five 



1. Systems analysis facility programs on the basis of client 

flow. 

2. Of the two types of systems, only the system is of interest to 

program evaluators. 

3. and are two dynamics 

that systems exhibit. 

4. Five processes affect the flow of clients through a system. They are: 
s » j , and . 

5. is the variation of client input and output. 

6. Systems analysis involves the , , and 

of relationships between program units. 

7. Four types of visual representations are regularly used to analyze 

systems. They are: charts, charts, 

charts, and charts. 

8. allows administrators to evaluate present system 

functioning and infer how changes in system components will effect 
programs. 

9. Six information units are needed to completely analyze a system. They 



are: rates, and rates, 

and rates, program rates, , and 

list figures. 

10. Flow analysis models are used to: the . 



of the facility, that need to be made, 

system components, and the effect of system 

11. can also be used to demonstrate the 

feasibility of initiating new services. 
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SELF-TEST: Chapter Five 
ANSWER KEY 



1. 
2. 

3. 

4. 

5. 
6. 

7. 

8. 
9. 

10. 

11. 
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Systems analysis depicts facility programs on the basis of client flow. 

Of the two types of systems, only the open system is of interest to 
program evaluators. 

Feedback loops and feedback delays are two dynamics that systems ex- 
hibit. 

Five processes affect the flow of clients through a system. They are: 
oscillation, routing, reservoirs, trajectory, and valves. 

Oscillation is the variation of client input and output. 

Systems analysis involves the definition^depiction^and examination of 
relationships between program units. 

Four types of visual representations are regularly used to analyze 
systems. They are: activity charts, layout charts, personal relationship 
charts, and data charts. 

Flow analysis allows administrators to evaluate present system function- 
ing and infer how changes in system components will effect programs. 

Six information units are needed to completely analyze a system. They 
are: intake rates, routing rules and rates* exit ports_^ind rates, pro- 
gram tenure rates, capacities, and waiting list figures. 

Flow analysis models are used to: illustrate the present operation of 
the facility, predict changes that need to be made, balance system 
components, and assess the effect of system changes. 

Systems analysis can also be used to demonstrate the feasibility of 
initiating new services. 
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Chapter Six 
Evaluation Research 



Even though ongoing data is gathered faithfully by facility program 
staff, some evaluation questions will require the use of more traditional 
research designs to interpret the collected data and provide answers. It 
may not be sufficient to simply know what happened and outcome data can 
only provide information on what happened. 

Of paramount concern for any rehabilitation administrator is the need 
to tie program outcomes, easily measured by ongoing data, to program 
activities. For example, if a person is successfully placed into a competi- 
tive environment following vocational training, can we, in fact, attribute 
this competitive success to the trailing? It may be possible to use existing 
outcome data with traditional research design to show that the success can 
be attributed to the program. 

Three tactics, mentioned earlier in Chapter Four, for establishing facts 
upon which conclusions can be drawn are normally employed by evaluation 
researchers. They are: case study, correlation, and experimentation. 



Case Studies 

Most existing data, especially data gathered for outcome evaluations, 
are found within client case files. Especially when evaluating programs 
after their completion (ex post facto studies) the only data that may be 
gathered must be gleaned from the existing case records. 

Data based on case study has several limiting factors. First, the 
questions that are asked by the evaluators must conform to the statistical 
data that was collected. For example, an evaluator could not pose the 
research question "What is the mean income of clients placed competitively 
after receiving training in the Services Training Program," if the case files 
do not contain any data about the client's income following placement. 
Second, the data that is entered into a case file may be "contaminated" by 
the number of staff allowed to enter data, varying definitions of data 
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points between staff and over time, and simple typographical errors. Fi- 
nally, accurate comparisons between groups may be difficult to establish. 

Even with the above limitations, case studies are a popular method of 
evaluation research in rehabilitation facilities simply because many evalua- 
tions are not pre-planned. By formulating evaluations after the fact, 
program administrators force evaluation researchers to use case study 
techniques. Planned evaluations will also use data that eventually becomes 
part of the client's case file. However, the planning process allows data to 
be gathered that clearly leads to answers for desired research questions. 
Case study data may be randomly selected, but is not usually used to sup- 
port an experimental design because of the difficulties in controlling vari- 
ables. 



Correlational Design 

A correlation is the determination that a direct relationship between 
variables exists. Correlations, however, cannot provide data about why the 
relationships exist. For example, if we find a correlation between atten- 
dance in a training program with competitive placement, we may infer that 
the correlation provides factual data supporting the belief that the program 
is effective. However, the correlation does not in fact show that the 
training itself caused competitive placement. Many other factors, such as 
the selection process or the placement efforts, may have influenced the 
outcome. 

Even with the above limitations, correlational design is often used to 
provide factual data for evaluators. Where true experimental designs using 
control groups and limited variables are not possible or may be too costly, 
a correlational study can provide a useful alternative or pilot study. In 
addition, the correlational study provides valuable descriptive data for use 
in both outcome and summative evaluations. Correlational data may be 
randomly selected and may, in fact, be used to support an experimental 
design. 



Experimental Design 

In true experiments a hypothetical "answer" to an evaluation question 
is "tested" by sampling controlled variable data that has been randomly 
selected. It is important that both random selection and variable control be 
included in the design to make the data collected by the experiment valid. 

Random selection has been discussed above. Variable control simply 
requires the evaluation researcher to discover and account for the depen- 
dent and independent variables. Dependent variables are the qualitative 
outcomes. Independent variables are the prefacing actions prior to a quali- 
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tative outcome. For example, competitive placement status after training 
may be the dependent variable with the various training options or cur- 
riculums within the options serving as independent variables. 

Experimental designs are often avoided by program evaluators because 
of the difficulties encountered in controlling variables, the time and ex- 
pense of conducting the research, and because ethical questions regarding 
the with-holding of training options are involved. However, experimental 
designs can be employed to many degrees within rehabilitation facilities. 



Traditional Research Design 

Traditional design relies on several levels of experimental research 
methods that incorporate the above research tactics. The evaluator must 
choose the level of design that best balances their need for valid data with 
their data collecting capabilities. Three traditional designs are usually 
cited. They exist not as discrete entities, but as levels on a continuum. 
These designs are pre-experimental, quasi-experimental, and true experi- 
mental designs. Pre-experimental models can simply be called outcome or 
descriptive evaluations because the intent is only to describe a program not 
to infer correlation or causation. The quasi-experimental and true ex- 
perimental models, however, try to establish correlations and infer causation 
between the variables under study. They vary only in their ability to 
control the variables with the true experimental model providing as close to 
full control over variables as possible. 



Quasi-experimental Models 

Many different quasi-experimental models can be created to meet the 
needs of various facility data collection problems and evaluation question 
needs. Three types are in common use. They are the single group pre/post 
test model, the nonequivalent control group pre/post test model, and the 
interrupted time seiies testing model. 



Single Group Pre/Post Test Model 

As the name implies, this research design is used when no control 
group is possible. Depending upon the type of information desired, the 
single group is first tested (data collected in some form) prior to their 
introduction to a training regime. Following training an identical testing 
(or data collection methodology) is applied to the group. Inferences are 
then based upon a comparison of the data from each testing. Note that the 
pre/post test may be an actual test, for example a skill test or productivity 
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test, or it may simply be the collection of descriptive data. The type of 
data collected is not important* That the same data is collected at both 
points is important 



Nonequivalent Control Group Pre/Post Test Model 

Just as in the single group pre/post test model, the nonequivalent 
control group model requires data collection at two points, before training 
and after training. In this model, however, another group is added as a 
"control." The control group is tested as if they were to receive training 
and then tested again as if they had received training, however, no training 
is provided. Thus, such data as the mean increase (or decrease) in test 
scores between the two groups can be compared. This provides yet another 
indication that training was successful (or unsuccessful). While more pow- 
erful than the single group model, the nonequivalent group is still vul- 
nerable to bias on the basis of the way the groups are selected. 



Interrupted Time Series Model 

Single or multiple groups can be exposed to the interrupted time series 
model. In this model, a group(s) is tested several times before and after 
providing training. By giving several pre and post tests and plotting the 
results, a linear relationship between the testing times can be established. 
The shape of the line helps the evaluator determine if increases (or de- 
creases) in test scores can be attributed to program intervention or some 
other factor. 
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EXERCISE NINE: Design an Evaluation Study 



To answer the program evaluation question below, describe how you 
would design an evaluation using any evaluation research design: experi- 
mental, quasi-experimental (single group pre/post test, nonequivalent group 
pre/post test, interrupted time series) correlational, or case study. 



The administrator of Rehabilitation Services, Inc. has asked you to 
design a study to help establish that the pillow-making work station pro- 
vides competitive skill training. 

The administrator has stated your null hypothesis as: Trainees enter- 
ing the pillow-making work station will show no increase in productivity 
before discharge or twelve weeks of training. 
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SELF-TEST: Chapter Si* 



L data can only provide information about what happened in 

the facility. They cannot provide information about or 

outcomes occurred. 

2. Administrators use evaluation research to tie program to 

program . 

3. Three tactics for establishing evaluation research data are: 

studies, studies, and . 

4. are based on the ongoing collection of data and, thus, 

research questions can be constructed after the data has "aged." 

5. studies attempt to find direct relationships between 

program outcomes and program activities. 

6. Experimental studies test collected data using controlled 

to provide data that allows evaluators to accept or 

reject evaluation questions. 

7. variables are the qualitative outcomes of the program 

or process being evaluated. 

8. variables are the prefacing actions, services, or ac- 
tivities that lead to qualitative outcomes. 

9. Three types of quasi-experimental models are useful for evaluating 

rehabilitation programs. They are: the 

pre/post test, the 

pre/post test, and the , 
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SELF-TEST: Chapter Six 
ANSWER KEY 



1. Outcome data can only provide information about what happened in the 
facility. They cannot provide information about how or why outcomes 
occurred. 

2. Administrators use evaluation research to tie program outcomes to pro- 
gram activities. 

3. Three tactics for establishing evaluation research data are: qase studies, 
correlational studies, and experiments. 

4. Case studies are based on the ongoing collection of data and, thus, re- 
search questions can be constructed after the data has "aged." 

5. Correlational studies attempt to find direct relationships between pro- 
gram outcomes and program activities. 

6. Experimental studies test randomly collected data using controlled vari- 
ables to provide data that allows evaluators to accept or reject hypo- 
thetical evaluation questions. 

7. Dependent variables are the qualitative outcomes of the program or 
process being evaluated. 

8. Independent variables are the prefacing actions, services, or activities 
that lead to qualitative outcomes. 

9. Three types of quasi-experimental models are useful for evaluating 
rehabilitation programs. They are: the single group pre/post test, the 
noneauivalent control group pre/post test, and the interrupted time 
series. 
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Chapter Seven 
Data Analysis Techniques 



In every program evaluation situation, data, either outcome or situa- 
tionally related, must be analyzed, allowing program administrators to make 
inferences about the value of the program under examination. Because the 
data is most often numerically expressed (the alternative would be an 
evaluation that reports that a prcduct was created or delivered), the analy- 
sis is usually of a statistical nature. The easiest way to express the an- 
swers to program evaluation questions is to statistically test the truth or 
falsity of a statement related to the question. This is called hypothesis 
testing. 



Hypothesis Testing 

A hypothesis is a statement, theoretical because it is unproven, re- 
garding the relationship of variables under study. The task of data analysis 
is to statistically indicate the probability that the statement is true or 
false. Thus, a hypothesis must be capable of being either true or false. It 
is common to state a hypothesis in a negative statement called the "null M 
hypothesis. For example, the evaluators of a vocational training program 
may state the null hypothesis as "The correlation of mean scores on the 
janitorial skill training battery A after training will be less than or equal 
to zero to those prior to receiving training." 

Corresponding to the null hypothesis, a "positive" hypothesis is also 
formulated. Thus the hypothesis "The correlation of mean scores on the 
janitorial skill training battery A after training will be greater than zero to 
those prior to receiving training," is the only alternative if the null 
hypothesis is shown to probably be false. 

The formulation of a hypothesis is probably the single most critical 
pre-evaluation planning needed prior to gathering and analyzing data. The 
hypothesis will dictate the t/pe of data that is collected and the type of 
analysis that will be conducted. 




Techniques 



Many different types of analytic techniques can be used to examine 
program evaluation data. (See Figure Twenty-two.) Many require statisti- 
cal computer programs and sophisticated knowledge of data manipulation 
procedures. Others, however, can be simply performed using hand cal- 
culators. Three of the simple techniques, sign tests, simple linear regres- 
sion, and chi-square, are described below. 



FIGURE TWENTY-TWO: Standard Data Analysis Methods 



DISTRIBUTION TYPE 



DATA TYPE 



ANALYTIC TYPE 



Normal distributions 
Non-normal distribution 
Non-normal distribution 
Non-normal distribution 



Interval/ratio data 
Nominal data 
Ordinal data 
Interval data 



Linear Regression 
Chi square 
Sign test 
T-tests 



Sign Tests 

This is probably the simplest test to use when determining if post-test 
ratings tend to be greater than pre-test ratings. Thus, for use with pro- 
gram evaluation data, the sign test will quickly help an evaluator determine 
positive or negative tendencies. The sign test will not tell the evaluator 
how great a change occurs but will indicate that change has occurred. 
Evaluators need only a table of critical values to determine a significance 
level for their findings. 

First, the data from each case is listed together with pre-test and 
post-test ratings. Second, the evaluator determines if the post-test rating 
is greater or less than the pre-test rating and assigns each case a plus (+) 
or a minus (-). The number of plus ratings arc counted. Third, the evalu- 
ator also notes the number of ties and subtracts the tics from the total 
number of cases. Fourth, the evaluator determines the level of significance 
that is needed to accurately test the hypothesis. Fifth, using both the 
adjusted total number of cases and the desired level of significance, the 
evaluator consults a mathematical table of critical values for sign tests. 
Finally, comparing the critical value found in the table with the number of 
plus ratings, the evaluator decides whether to accept or reject the null 
hypothesis. 
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An Example 

To help establish the ability of the pillow-making work station in 
providing competitive skill training, the administrator of Rehabilitation 
Services, Inc. asked the supervisor of the pillow-making work station to 
provide: 1) A list of all clients entered into the work station during 19XX, 
2) Their productivity ratings after one week of orientation and procedures 
training, and 3) Their productivity ratings after 12 weeks in the program 
(or their rating at the time of discharge if they were placed competitively, 
self-terminated, or dismissed prior to the end of 12 weeks of training. 
Twelve weeks was the length of time that the facility had targeted for 
reaching competitive ratings.) 

The administrator determined the null hypothesis: Trainees entering 
the pillow-making work station will show no increase in productivity before 
discharge or twelve weeks of training. 



List cases with pre-test and post-test ratings. 

The work station supervisor provided the following data: 

CASE NUMBER PRODUCTIVITY PRODUCTIVITY RATING 

RATING AFTER AT DISCHARGE OR 
ONE WEEK TWELVE WEEKS 



1 


41 


48 


2 


15 


81 


3 


67 


55 


4 


73 


94 


5 


78 


85 


6 


53 


51 


7 


69 


61 


8 


40 


63 


9 


70 


81 


10 


07 


97 


11 


26 


25 


12 


70 


77 


13 


23 


23 


14 


33 


41 


15 


61 


26 


16 


46 


60 


17 


12 


49 


18 


•79 


84 


19 


>3 


88 


20 


40 


67 
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Analyze the data 



The first step in analyzing the sign test data is to determine if the 
post-training data is greater than the prc-training data. If the post- 
training rating is larger than the pre-training rating, a plus (+) is assigned 
to the case; if it is smaller, a minus (-) is assigned. The total number of 
"plus" signs is tallied as well as the number of tics. The tics arc sub- 
tracted from the total number of cases. (Sec below.) 



CASE NUMBER PRODUCTIVITY 
RATING AFTER 
ONE WEEK 



1 


41 


2 


15 


3 


67 


4 


73 


5 


78 


6 


53 


7 


69 


8 


40 


9 


70 


10 


07 


11 


26 


12 


70 


13 


23 


14 


33 


15 


61 


16 


46 


17' 


12 


18 


79 


19 


83 


20 


40 



PRODUCTIVITY PLUS OR 

RATING AT MINUS 

DISCHARGE OR ASSIGNED 
TWELVE WEEKS 



48 


+ 


81 


+ 


55 




94 


+ 


85 


+ 


51 




61 




63 


+ 


81 


+ 


97 


+ 


25 




77 


+ 


23 


0 


41 


+ 


26 




60 


+ 


49 


+ 


84 


+ 


88 


+ 


67 


+ 



- 1 14 ■■+« 

1 Tie 

N = 19 5 «»-■■ 



Compare data to critical values. 

Now that the number of M + H assignments and tics have been deter- 
mined, the evaluator determines the level of significance desired. This 
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statistic indicates the probability that the test will reject the null hypoth- 
esis when it is actually correct. Two levels arc usually considered, the .05 
level and the .01 level. For most applications, the .05 level of significance 
(the test will accurately reject the null hypothesis 95% of the time) is 
sufficient. In this example, we will assume that the administrator is com- 
fortable with the .05 level. All that remains is to determine the critical 
value for 19 cases at the .05 significance level. This is accomplished by 
consulting a binomial distribution table (see Figure Twenty-three.) 



FIGURE TWENTY-THREE: Critical Values of One-Sided Sign Test 



CASES .05 SIG. .01 SIG. 



5 


5 


5 


6 


6 


6 


7 


6 


7 


8 


7 


8 


9 


8 


9 


10 


9 


10 


11 


9 


10 


12 


10 


11 


13 


10 


12 


14 


11 


12 


15 


12 


13 


16 


12 


14 


17 


13 


14 


18 


13 


15 


19 


14 


15 


20 


15 


16 



Reading across the case column at 19 to the .05 significance column, 
we obtain a critical value of 14. If the actual statistic is equal to or 
greater than the critical value we will be forced to reject the null hypothe- 
sis. Because 14 plus signs were found in our data we reject the null 
hypothesis that states that no change will be found. Therefore, we con- 
clude that their is a significant possibility that a change in productivity 
occurred between the prc-training rating and the post-training rating. We 
can, with reservation, say that the program may have had a positive affect. 
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Linear Regression 



Some data may lend themselves to analysis by trend line forecasting. 
This is particularly helpful when making predictions about client progress, 
but can also be used to illustrate program and/or facility directional 
change. In fact, change rate may also be of interest to program evaluators. 



Least-Squares Forecasting 

One of the easiest and most common linear regression techniques is 
the "least squares" forecasting method. This method is relatively simple in 
its use of mathematics and provides good trend line information. The 
method used is a statistical manipulation of data points. All of the squares 
(a number ^multiplied by itself) of these data points are added and reformu- 
lated mathematically. The resulting new data can be plotted to illustrate 
the differences between the statistically determined figures and the actual 
data. This graphic representation of the trend line data (see Figure Twen- 
ty-four) represents the least amount of difference between those data 
points above the line and those below the line. The line, therefore, can be 
used to predict, with reasonable accuracy, data for points in the near 
future. 



FIGURE TWENTY-FOUR: Sample Least Squares Regression Line 
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The linear regression forecasting method employed by the least squares 
method cannot separate cyclical or seasonal effects on the points plotted on 
the demand graph. Thus, the data must be "deseasonalized" to give an 
accurate forecast of trends. (Some program evaluation data points may not 
be affected by seasonal or cyclic trends. Deseasonalization may not be 
necessary in that case or in the case of data that is stated in yearly 
terms.) 

The following steps are used for creating data to use in least squares 
forecasting: 

1. Obtain enough historical data to support a regression analy- 
sis. If forecasts are to be made with the trend line, twelve 
data periods (usually months) is usually enough to forecast up 
to twelve more periods of future data. More data points 
provide a better forecast. Less can be used, but the results 
may not be reliable. 

2. Insure that the time increments within the data are incre- 
mental. That is, each period of time that the data repre- 
sents should be equal to all the other periods. If you are 
gathering daily work data, remember that the number of 
work days in a month will vary. Thus, if the daily data is 
reflected in a monthly figure, you must adjust the periods to 
reflect the number of days that are actually worked. 

3. Adjust the periods to a common point in time if units may 
change over time (such as dollars earned). Inflation can give 
a false picture within a forecast. To guard against bias, 
relate all periods to a common year. 

4. Deseasonalize the data. Because some data figures may be 
used on a basis that fluctuates seasonally, the data needs to 
be adjusted to account for these fluctuations if an accurate 
forecast is to be made. 

5. Plot the adjusted data on a time vs. quantity graph. 

6. Using the seasonally adjusted data and the least squares 
regression formulas, determine the trend line existing for the 
plotted data. 

7. Examine the trend line to make predictions about future data. 
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An example 



We will use the following records of mean monthly wages in the 
pillow-making work station at Rehabilitation Services, Inc. to provide an 
example of least squares regression forecasting. 



The work station supervisor is asked to examine all of the production 
records during the past year to determine, on a monthly basis, the mean 
hourly wage rates paid to the station's workers. The supervisor submits 
the following list: 



Because the pillow making work station production reports reflected 
the wages paid on a monthly basis without regard to the number of actual 
working days in the month, we must first adjust the data to compensate for 
months with high and low numbers of working days. Monthly wages are 
first divided by the number of work days in the month. They are then 
multiplied by the average number of work days in each month, in this case 
21, to arrive at the monthly adjusted wages. See Figure Twenty-five below: 



Obtain historical data 



Hourly wage rates: 19XX 



January 

February 

March 

April 

May 

June 

July 

August 

September 

October 

November 

December 



47,50 
62.50 
55.00 
67.50 
57.50 
70.00 
77.50 
87.50 
92.50 
62.50 
50.00 
42.50 



Adjust time periods to a common base 
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FIGURE TWENTY-FIVE: Sample Time Period Adjustment 



Wages per Average Adjusted 
Mean Working working Days Mean 
Month Wages Days Day Per Month Wages 



January 


47 


.50 : 


• 


22 




2 


.16 


X 


21 




45 


.36 


February 


62 


.50 • 




20 




3 


.13 


X 


21 




65 


.73 


March 


55 


.00 • 




19 




2 


.89 


X 


21 




60 


.69 


April 


67 


.50 • 




22 




3 


.07 


X 


21 




64. 


.47 


May 


57 


.50 ■ 




21 




2 


.74 


X 


21 




57. 


.54 


June 


70 


.00 ■ 




21 




3 


.33 


X 


21 




69. 


.93 


July 


77 


.50 i 




23 




3 


.37 


X 


21 




70. 


.77 


August 


87 


.50 • 




21 




4 


.17 


X 


21 




87. 


.57 


September 


92 


.50 ; 




21 




4 


.40 


X 


21 




92. 


.40 


October 


62 


.50 ■ 




23 




2 


.72 


X 


21 




57. 


12 


November 


50 


.00 ■ 




18 




2 


.78 


X 


21 




58. 


38 


December 


42 


.50 ; 




21 




2 


.02 


X 


21 




42. 


42 



252 



252 -J- 12 = 21 



Adjust periods to common dollars 

In this example, the wages are being forecast on the basis of dollars 
in one year. The influences of inflation will not be a major factor in the 
variation. If inflation were a major factor, all of the dollar amounts would 
have been adjusted to a base year by multiplying all amounts by an infla- 
tion factor obtained from reports provided by the Government Printing 
Office. 



Dese a sonalize the data 

A quick examination of the wages paid tells the program evaluator 
that seasonal factors may be involved in the production of pillows and 
reflected in the wages paid during the month. Late summer and early fall 
increases show the effect of the major purchaser of the facility's output, 
Pillowraatic, attempting to stock stores for holiday purchases. Therefore, to 
determine the true wage trend, the data must be deseasonalized. 

To deseasonalize the data the adjusted wages per month are summed 
end divided by 12 to find an average for the adjusted data points. The 
adjusted monthly wages for each month are now divided by the average to 



obtain each month's seasonal index. This seasonal index is now multiplied 
by the adjusted wages per month to obtain a deseasonalized figure for use 
in examining the data trends. See Figure Twenty-six below. 



FIGURE TWENTY-SIX: Sample Seasonal Adjustment 



Average S ea s ona 1 ly 

Adjusted Adjusted Seasonal Adjusted Adjusted 



Month 


Wacres 






Waqes 




Index 




V7aaes 




Waae 


S 


January 


45. 


36 






64. 


365 




.705 


X 


45. 


36 




32. 


00 


February 


65. 


73 






64. 


365 




1.021 


X 


65. 


73 




67. 


00 


March 


60. 


69 






64. 


365 




.943 


X 


60. 


69 




57. 


00 


April 


64. 


47 






64. 


365 




1.002 


X 


64. 


47 




65. 


00 


May 


57. 


54 






64. 


365 




.894 


X 


57. 


54 




51. 


00 


June 


69. 


93 






64. 


365 




1.086 


X 


69. 


93 




76. 


00 


July 


70. 


77 






64. 


365 




1.100 


X 


70. 


77 




78. 


00 


August 


87. 


57 






64. 


365 




1.361 


X 


87. 


57 




119. 


00 


September 


92. 


40 






64. 


365 




1.436 


X 


92. 


40 




133. 


00 


October 


57. 


12 






64. 


365 




.887 


X 


57. 


12 




51. 


00 


November 


58. 


38 




r 


64. 


365 




.907 


X 


58. 


38 




53. 


00 


December 


42. 


42 






64. 


365 




.657 


X 


42. 


42 




28. 


00 




772. 


38 






















810. 


00 



Plot the adjusted data 

Now that the monthly data has been adjusted for both the number of 
working days in the month and seasonal fluctuations, we will graphically 
illustrate the data. See Figure Twenty-seven below: 
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FIGURE TWENTY-SEVEN: Sample Adjusted Data Plotting 
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60.00 



30.00 

















































































i 


























< 


> 
























































i 




















i 




i 


> 


< 


► 




























► 








i 


> < 


1 








< 


> 


















































i 


i 





































JAN FEB MAR APR HAY JUN JUL AUG SEP OCT NOV DEC TIME 



Calculate a line of best fit 

We are now ready to determine the regression line (line of best fit) 
for the data points. The easiest way to handle the data for determining 
this line is to create a graph of all the relevant data including time and 
the adjusted data points from above. See Figure Twenty-eight below. 
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FIGURE TWENTY-EIGHT: Gathering Least Square Forecast Data 



X 

(TIME) 


X* 


Y 

(WAGES) 


X'Y 


2 

X 1 


1 










2 










3 










4 










5 










6 










7 










8 










9 










10 










11 










12 























N « 




E X 1 


E Y 


E X 1 Y 


2 

E X 1 



Determine X' 

In order to manipulate the data using the least squares method, we 
must transform the time increments into positive and negative figures. 
Each month will be assigned a number from one to twelve in succession. 
January is One, February is Two, etc. These numbers are then added, 
resulting in a figure of 78. Because there are 12 months, 78 divided by 12 
finds a midpoint of 6.5. Therefore, X* (X prime) is assigned by giving 
each month above 6.5 a number sequentially from 1 through 6 and each 
month below 6.5 a number sequentially from -1 through -6. See Figure 
Twenty-nine. (If the data you are using has an odd number of data 
points, the midpoint should be labeled zero.) The sum of these numbers (E 
X') must be zero. 
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Determine Y value 



The third column in Figure Twenty-nine is labeled "Y". This column is 
reserved for the adjusted data on the wages paid each month. Note that 
the sum of all the data points (E Y) is also determined. 

Determine X'Y 

The fourth column in Figure Twenty-nine is labeled "X'Y". This 
column holds the product of the multiplication of the assigned X* time value 
and the wages paid (adjusted for monthly and seasonal variations). These 
figures will be both positive and negative depending upon the assigned X 1 
value. The sum of this column is also determined. It is labeled EX'Y. 



The final column of Figure Twenty-nine holds the squares of the 
assigned time values (X*). When negative numbers are squared, they become 
positive. Therefore, -6 x -6 « 36. This column is also summed. 



Determine X' squared 
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FIGURE TWENTY-NINE: Sample Least Squares Data 



X 

(TIME) 






Y 

(WAGES) 


X'Y 


2 

X' 


1 




-6 


32.00 


- 192.00 


36 


2 




-5 


67.00 


- 335.00 


25 


3 




-4 


57.00 


- 228.00 


16 


4 




-3 


65.00 


- 195.00 


9 


5 




-2 


51.00 


- 102.00 


4 


6 




-1 


76.00 


- 76.00 


1 


7 




1 


78.00 


78.00 


1 


8 




2 


119.00 


238.00 


4 


9 




3 


133.00 


399.00 


9 


10 




4 


51.00 


204.00 


16 


11 




5 


53.00 


265.00 


25 


12 




6 


28.00 


168.00 


36 



78 




0 


810.00 


224.00 


182 


N = 12 




E X' 


E Y 


E X 1 Y 


2 

E X' 



Calculate points on line of best fit 

All of the data has now been determined to allow the line of best fit 
to be located. Each point is determined by using the following formula: 



E Y + X' x E X' Y = Point on line 

N 2 of best fit 

E X« 
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It is not necessary to determine all twelve points in order to draw the 
line on our graph. We need only to determine the points for January (I) 
and December (12). Therefore we insert the figures for January and 
December in the formula: 



810 
12 



[ -6 X 
[ 



224.00 ] 
182 ] 



60.12 



810 
12 



[ 6 X 224.00 ] 
[ 182 ] 



74.88 



These points are added to our graph and a line is drawn connecting 
them. This line represents the line of best fit and will be used to forecast 
monthly wage rates for the coming year. See Figure Thirty below. 



FIGURE THIRTY: Sample Least Squares Forecasting Line 
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Use line of best fit to forecast mean monthly wages 

Extending the line of best fit out on the graph for twelve more equal 
monthly increments, wc can estimate the mean wage rates in the pillow- 
making work station for the coming year. Remember, however, that the 
line is premised on wages adjusted for seasonal variations. Thus, to know 
the projected mean wages for any given future month, we must remove the 
seasonal adjustment. This is accomplished by building a chart of projected 
mean wages and multiplying the projections for each month by the seasonal 
index (used above). See Figure Thirty-one below. 



FIGURE THIRTY-ONE: Sample Mean Wage Projections 











Seasonally 


Wage 


Pro j ected 


Forecast 




Seasonal 


Adjusted 


Running 


Month 


Mean Waaes 




Index 


Mean Wacres 


Total 


January 


76.00 


X 


.705 




53.58 


53.58 


February 


77.50 


X 


1.021 


S3 


79.13 


132.71 


March 


79.00 


X 


.943 


ss 


74.50 


207.21 


April 


81.00 


X 


1.002 




81.16 


288.37 


May 


82.50 


X 


.894 




73.76 


362.13 


June 


84.00 


X 


1.086 




91.22 


453.35 


July 


85.00 


X 


1.100 


EZ 


93.50 


546.85 


August 


87.00 


X 


1.361 




118.41 


665.26 


Septesiber 


87.50 


X 


1.436 




124.93 


790.19 


October 


89.00 


X 


.887 




78.94 


869.13 


November 


90.00 


X 


.907 




81.63 


950.76 


December 


92.00 


X 


.657 




60.44 


1011.20 



Chi-square Tests 

The task of a program evaluation may be to determine relationships 
between different classes of variable?;. Two primary variations are normally 
found. In the simplest variation, the data may be represented by a 2 x 2 
grid. In more complex situations the grid may contain many cells. 



Simple Chi-squares 

A common relationship that is explored in program evaluation is the 
relationship between successful training closure and the obtaining of com- 
petitive employment. The grid for analyzing such a relationship may look 
like: 
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Competitively 
Employed 

YES 



NO 



Successfully completed 
training 
YES NO 



A 


B 


C 


D 



Rehabilitation Services, Inc. was interested in the relationship of 
successful training and competitive employment The administrator of the 
facility formed the null hypothesis that a chi-squarc analysis would indicate 
that no difference was found between the number of clients successfully 
completing training and arc competitively employed and those who do not 
complete training and arc competitively employed. During the course of 
19XX their twelve work stations served and discharged 138 clients. (More 
were served but remained in the training program at the time of this 
program evaluation). After making a list of all clients that had been dis- 
charged during the year for any reason, the supervisor tallied the number 
who had been discharged following successful completion of the program 
and those who had cither been dismissed or had self-terminated. Eighty- 
five clients had been discharged following successful completion of training; 
fifty-three did not complete training successfully. Because all discharged 
clients, both succcssfuJly and unsuccessfully discharged, received follow-up 
services through the facility's case manager's office, most of the client files 
included notations regarding their current employment status. The case 
managers were asked to update those files that did not include this data. 
When all the files were complete, the supervisor tallied those clients who 
reported holding at least half-time competitive jobs. Seventy-nine former 
clients reported that they held at least half-time competitive positions; 
fifty-nine were unemployed, had jobs that were less than half-time, or 
worked in sheltered or voluntary situations. 

The supervisor also paired the client data, tallying those who success- 
fully completed training and held a competitive job, those who successfully 
completed training and did not hold a job, those who did not complete 
training and held a competitive job, and those who did not complete train- 
ing and did not hold a competitive job. This data was used to fill out the 
following grid: 
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Successfully completed 
training 



Competitively 
Employed 

YES 



NO 



YES 



NO 



A 

58 


B 

21 


C 

27 


D 

32 



A+C=85 



B+D=53 



A+B=79 

C+D=59 
N=138 



To analyze this data, the administrator must determine both a critical 
value (which will be located in a chi-squarc table of critical values) and a 
test statistic for the data to be analyzed. 

The test statistic is determined using the following formula: 

2 

2 N (AD-BC) 

X 



(A+B) (C+D) (A+C) (B+D) 



Thus, replacing the letters with the figures from the grid above, the 
administrator determines that the test statistic is 10.92: 



2 

2 138 X [ (58 X 32) - (21 X 27) ] 

X = 

79 X 59 X 85 X 53 



2 229,289,898 

X = - 10.92 

20,997,805 
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Critical values in chi-squarc tables arc listed by "degrees of freedom." 
This statistic is determined by multiplying the number of rows in your data 
grid less one by the number of columns in the grid less one. Whenever 
your data conforms to the 2 X 2 pattern above, the degrees of freedom will 
be 1 because the number of rows (2) minus 1 « l t multiplied by the number 
of columns (2) minus 1 « 1, results in a product of 1 (1 x 1 = 1.) (Note: 
as in the other tests, a significance level will need to be determined. For 
most program evaluation analyses, a significance level of .05 is sufficient. 
A level of .01 is very significant and should be reported if possible.) 

Consulting a table of critical values for chi-squarc, sec Figure Thirty- 
two, the administrator determines that the critical value at the .05 level 
with one degree of freedom is 3.84. The value at the .01 level is found to 
be 6.64. Because the test statistic of 10.92 is greater than cither of thec, 
the null hypothesis is rejected. Thus, successful completion of a Rehabili- 
tation Services, Inc. training program is correlated with obtaining a com- 
petitive job. 
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FIGURE THIRTY-TWO: Critical Values of Chi-sqi are. 



d.f. 


a » .05 


a « .01 


d.f. 


a = .05 


tt » .01 


d.f. 


a = .05 


a = .01 


1 


3.841 


6.635 


46 


62.830 


71.201 


91 


114.268 


125.289 


2 


5.991 


9.210 


47 


14.001 


72.443 


92 


115.390 


126.462 


3 


7.815 


1 ! .345 


48 


65.171 


73.683 


93 


116.511 


127.633 


4 


9.458 


13.277 


49 


66.339 


74.919 


94 


117.632 


128.803 


5 


11.071 


15.086 


50 


67.505 


76.154 


95 


118.752 


129.973 


6 


12.592 


16.812 


51 


68.669 


77.386 


96 


119.871 


131.141 


7 


14.067 


18.475 


52 


69.832 


78.616 


97 


120.990 


132.309 


8 


15.507 


20.090 


53 


70.993 


79.843 


98 


122.108 


133.476 


9 


16.919 


21.666 


54 


72.153 


81.069 


99 


123.225 


134.642 


10 


18.307 


23.209 


55 


73.311 


82.292 


100 


124.342 


135.807 


11 


19.675 


24.725 


56 


74.468 


83.513 


102 


126.574 


138.134 


12 


21.026 


26.217 


57 


75.624 


84.733 


104 


128.804 


140.459 


13 


22.362 


27.688 


58 


76.778 


85.950 


106 


131.031 


142.780 


14 


23.685 


29.141 


59 


77.931 


87.166 


108 


133.257 


145.099 


15 


24.996 


30.578 


60 


79.082 


88.379 


110 


135.480 


147.414 


16 


26.296 


32.000 


61 


80.232 


89.591 


112 


137.701 


149.727 


17 


27.587 


33.409 


62 


81.381 


90.802 


114 


139.921 


152.037 


18 


28.869 


34.805 


63 


82.529 


92.010 


116 


142.138 


154.344 


19 


30.144 


36.191 


64 


83.675 


93.217 


118 


144.354 


156.648 


20 


31.410 


37.566 


65 


84.821 


94.422 


120 


146.567 


158.950 


21 


32.671 


38.932 


66 


85.965 


95.626 


122 


148.77? 


161.250 


22 


33.924 


40.289 


67 


87.108 


96.828 


124 


150.989 


163.546 


23 


35.172 


41.638 


68 


88.250 


98.028 


126 


153.198 


165.841 


24 


36.415 


42.980 


69 


89.391 


99.228 


128 


155.405 


168.133 


25 


37.652 


44.314 


70 


90.531 


100.425 


130 


157.610 


1 TA /111 


26 


38.885 


45.642 


71 


91.670 


101.621 


132 


159.814 


172.711 


1!7 


40.113 


46.963 


72 


92.808 


102.816 


134 


162.016 


174.996 


28 


41.337 


48.278 


73 


93.945 


104.010 


136 


164.216 


177.280 


29 


42.557 


49.588 


74 


95.081 


105.202 


138 


166.415 


179.561 


30 


43.773 


50.892 


75 


96.217 


106.393 


140 


168.613 


181.840 


31 


44.985 


52.191 


76 


97.351 


107.583 


142 


170.809 


J 84.1 18 


32 


46.194 


53.486 


77 


98.484 


108.771 


144 


173.004 


186.393 


33 


47.400 


54.776 


78 


99.617 


109.958 


146 


175.198 


188.666 


34 


48.602 


56.061 


79 


100.749 


111.144 


148 


177.390 


190.938 


35 


49.802 


57.342 


80 


101.879 


112.329 


150 


179.581 


193.208 


36 


50.998 


58.619 


81 


103.010 


113.512 


200 


233.994 


249.445 


37 


52.192 


59.892 


82 


104.139 


114.695 


250 


287.882 


304.940 


38 


53.384 


61.162 


83 


105.267 


115.876 


30u 


341.395 


359.906 


39 


54.572 


62.428 


84 


106.395 


117.057 


400 


447.632 


468.724 


40 


55.758 


63.691 


85 


107.522 


118.236 


500 


553.127 


576.493 


41 


56.942 


64.950 


86 


108.64S 


119.414 


600 


658.094 


683.516 


42 


58.124 


66.206 


87 


109.773 


120.591 


700 


762.661 


789.974 


43 


59.304 


67.459 


88 


110.898 


121.767 


800 


866.911 


895.984 


44 


60.481 


68.710 


89 


112.022 


122.942 


900 


970.904 


1001.630 


45 


61.656 


69.957 


90 


113.145 


124.116 


1000 


1074.679 


1106.969 



From: Bradley, James V. (1968). DISTRIBUTION-FREE STATISTICAL TESTS. 
Englewood Cliffs, N.O.: Prentice-Hall. P. 379. 
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Complex Chi-squares 



A more complex program evaluation question may lead to a more 
elaborate grid structure. For example, we may wish to show that all of our 
supported work sites provide training for a cross section of severe dis- 
abilities, ^he grid for analyzing such relationships may look like: 



Work Severe disability type 

Station MR/DD MI VISUAL PARA QUAD 



SEARS 



HOLIDAY INN 



TARGET 



The administrator of Rehabilitation Services, Inc. was concerned about 
accusations that some community members were making that the facility was 
not providing equal supported work opportunities to severely disabled clients 
in the various supported work sites developed by the facility. Discussing 
the situation with the supervisor of the support work program, the ad- 
ministrator decided to test the null hypothesis that there is a relationship 
between disability type and placement in one of the three supported work 
sites that the facility operates. 

The facility served 74 clients with five disabling conditions in three 
supported work sites. Thus, the administrator asked the supported work 
program supervisor to complete the following grid by identifying the number 
of clients served in each site during the past year according to their 
primary disability classification: 



Work Severe disability type 

Station MR/DD MI VISUAL PARA QUAD 



SEARS 


9 


8 


5 


8 


4 


34 


HOLIDAY INN 


8 


9 


6 


3 


1 


27 


TARGET 


4 


2 


2 


3 


2 


13 




21 


19 


13 


14 


7 


N=74 
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Chi-square analysis is a test that compares the expectations of fre- 
quencies in rows and columns with the observed frequencies in rows and 
columns. The following formula is used, where "0" equals the observed 
frequency (actual data) and "E" equals the expected frequency: 



( 0 - E ) 



Expected frequencies are determined using the formula: 



Sum of the Row X Sum of the column 

E = 

Total sample size (N) 

For example, the grid cell corresponding to the row for Sears and the 
column for MR/DD has an observed frequency ("0") of 9 and an expected 
frequency of 9.6, The expected frequency was determined by multiplying 
the row total with the column total and dividing the product by the total 
sample size ( W N W ): 

34 X 21 

= 9.6 

74 

Using this procedure, the supervisor determined the expected frequen- 
cies for each cell of the grid (Observed/Expected): 



Work 
Station 

SEARS 

HOLIDAY INN 
TARGET 



Severe disability type 

MR/DD MI VISUAL 



PARA 



9 / 9.6 


8 / 8.7 


5 / 6.0 


8 / 6.4 


4 / 3.2 


8 / 7.7 


9 / 6.9 


6 / 4.7 


3 / 5.1 


1 / 2.6 


4 / 3.7 


2 / 3.3 


2/2.3 


3/2.5 


2 / 1.2 



QUAD 
34 



21 



19 



13 



14 



27 
13 
N=74 



146 

153 



To complete the formula for finding the test statistic, it is easiest to 
build a chart listing the observed frequencies with their associated expected 
frequencies, the result of subtracting the expected frequency from the 
observed frequency, that result squared, and the product of dividing the 
squared result by the expected frequency. The chart for our example is 
below: 



Observed 
Freq. 

0 


^ Expected < 
Frequency ] 

E 


ibserved minus 
Sxpected Freq 

O-E 


s Observed minu. 
. Expected Freq 
Squared 

2 

( O - E ) 


i 

s (O-E) divided 
. by Expected 
Frequency 

2 

( O - E ) 
E 




9 


9.6 


- .60 


.36 


.04 


o 
o 


o » / 


— *7fl 


AO 


• uo 


5 


6.0 


-1.00 


1.00 


.17 


8 


6.4 


1.60 


2.56 


.40 


4 


3.2 


.80 


.64 


.20 


8 


7.7 


.30 


.09 


.01 


9 


6.9 


2.10 


4.41 


.64 


6 


4.7 


1.30 


1.69 


.36 


3 


5.1 


-2.10 


4.41 


.86 


1 


2.6 


-1.6 


2.56 


.98 


4 


3.7 


• 30 


.09 


.02 


2 


3.3 


• -1.30 


1.69 


.51 


2 


2.3 


- .30 


.09 


.04 


3 


2.5 


.50 


.25 


.10 


2 


1.2 


.80 


.64 


.53 



2 

N » 74 X - 4.92 
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Completing the chart, the administrator adds all the figures in the 
right column to determine the chi-square statistic for this set of evaluation 
data. It is 4.92. 

To find the critical value from a chi-square table, like in the example 
above, we must first establish the degrees of freedom fcr the data. This is 
accomplished by multiplying the number of rows in our original data (3) 
minus 1, with the number of columns (5) minus 1. Thus we multiply 2x4 
and determine the number of degrees of freedom to equal 8. 

Consulting the chi-square tables (see Figure Thirty-two) the adminis- 
trator learns that the critical value for 8 degrees of freedom at a .05 
significance level is 15.51. Because the test statistic is below this figure, 
the null hypothesis must be accepted. Therefore, this test indicates that a 
relationship between disability type and placement within a supported work 
environment does exis and that the concerns of those critical to the 
facility may have a basis in fact. 
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EXERCISE TEN: Linear Regression Worksheet 



The Director of Rehabilitation Services, Inc. was concerned about the 
number of clients that were being referred to Tie placement program from 
vocational workstation four (shrink-wrapping). Because the numbers vary 
each month, the Director asks you to do a linear regression to find the 
trend line for these referrals. Use the following data and the least squares 
formulas to determine a projection line. 



Clients referred 
to the Placement 
Program from Voc 
Workstation Four 



January 

February 

March 

April 

May 

June 

July 

August 

September 

October 

November 

December 



7 
2 
5 
7 
7 
0 
7 
7 
2 
2 
0 
2 



Referral Average Ad j usted 
Mean Working per Working Days Mean 
Month Referrals Days Day Per Month Referrals 
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EXERCISE TEN 
CONTINUED 



Average Seasonally 
Adjusted Adjusted Seasonal Adjusted Adjusted 
Month Referrals Referrals Index Referrals Referrals 



9 
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EXERCISE TEN CONTINUED 



X 

(TIME) 


X 1 


y 

(WAGES) 


X'Y 


2 

X 1 


1 










2 










3 










4 










5 










6 










7 










8 










9 










10 










11 










12 























N - 




E X 1 


E Y 


E X 1 Y 


2 

E X 1 



Seasonally Referral 
Projected Forecast Seasonal Adjusted Running 
Month Referrals Index Referrals Total 
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EXERCISE ELEVEN: Chi-square Worksheet 



The Director of Rehabilitation Services, Inc. is writing a grant to 
obtain funds for the continued operation of a janitorial crew program. In 
the grant, the Director wishes to tie successful completion of the program 
with the client's competitive employment. The supervisor is asked to 
provide data about the clients and their competitive status. You are asked 
to use the data to determine if a relationship exists. You are to use a chi- 
square analysis. 



Entered the program = 32? 

Graduated from the program = 132 

Graduated and obtained employment = 97 

Did not graduate but obtained employment = 63 



Competitively 
Employed 

YES 



NO 



Successfully completed 
training 
YES NO 



A 


B 


C 


D 
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SELF-TEST: Chapter Seven 



1. Program evaluation data must be by end users or their 

representatives in order to make about the program 

on the basis of evaluation data. 

2. The task of data analysis is to indicate the probability 

that a hypothesis (a theoretical statement of the relationship between 
variables) is , 

3. Hypotheses are stated in terms. 

4. Three simple data analysis techniques are: s 



_j and 



5. are the simplest tests to use when determining if 

post-test ratings tend to be than pre-test ratings. 

6. Data forecasting using are most helpful when 

making predictions of future activities or illustrating positive or nega- 
tive trends. 

7. forecasting is a representation of the difference 



between data points above and below the average. 

8. To determine the relationships that exist between different classes of 
variables, a test may be most appropriate. 

9. Simple chi-squares involve only variables. 

10. The analysis of chi-square data requires the use of a 

table. 
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SELF-TEST: Chapter Seven 
ANSWER KEY 



1. 

2. 

3. 
4. 

5. 

6. 

7. 

8. 

9. 
10. 
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Program evaluation data must be analyzed by end users or their repre- 
sentatives in order to make inferences about the program on the basis 
of evaluation data. 

The task of data analysis is to statistically indicate She probability that 
a hypothesis (a theoretical statement of the relationship between vari- 
ables) is true or fal se. 

Hypotheses are stated in null terms. 

Three simple data analysis techniques a:e: sign tests, simple linear 
regression, and chi-sauars . 

Sign tests are the simplest tests to use when determining if post-test 
ratings tend to be greater than pre-test ratings. 

Data forecasting using linear regressions are most helpful when making 
predictions of future activities or illustrating positive or negative 
trends. 

Least-sQuares forecasting is a representation of the difference between 
data points above and below the average. 

To determine the relationships that exist between different classes of 
variables, a chi-souare test may be most appropriate. 

Simple chi-squares involve only two variables. 

The analysis of chi-square data requires the use of a critical value 
table. 




Chapter Eight 
Report Results 



Graphic displays will help make the presentation of program evaluation 
research data more meaningful. Often used graphics are: pie graphs, bar 
graphs, and line graphs along with many other types of visual aids. A 
narrative report will most likely accompany any visual representation of the 
evaluation data. These reports are often called "management reports," 
especially when conducting summative evaluations. 

It is important to include explanations of any data that is presented. 
When any of the analysis techniques discussed in chapter seven are used to 
give meaning to raw program evaluation data, the significance levels and 
means of the data should be reported along with the test statistics and 
critical values. It is not sufficient to state "The data indicate that there is 
a positive relationship of successful program completion to competitive 
placement." This statement must be qualified by stating "This conclusion 
was based on a chi-square statistic of 10.92 as compared with the critical 
value at one degree of freedom of 6.64 at the .01 significance level." 



The results for the current and cumulative program evaluation periods 
are compiled in the form of a "management report" that should include: 
the program results by objective, program results by client descriptors, a 
narrative report that summarizes the results, and the results of any special 
program evaluation questions. 

A good narrative report should be able to take the mystery and con- 
fusion out of the statistical reports so that either management or the board 
who are unfamiliar with the program evaluation process can fully under- 
stand how successful the agency was during the period in accomplishing its 
goals. They also need to understand the implications of recommendations 
made in light of the program evaluation results. Vocational evaluation 
programs have special needs concerning reports, especially reports of client 
evaluations. Soiae of the material included in the Thomas (1986) publication 



Report Formats 
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Report Writing in Assessment and Evaluation can be used in program evalu- 
ation situations as well. 

The narrative management report should include all of the agency's 
programs. There should be a separate narrative report section for each 
program. Significant developments should be highlighted in the report. 

Following is a sample report of an evaluation of Rehabilitation Ser- 
vices, Inc.'s vocational programs and services for the month of March, 
198X, and the cumulative period January 1, 198X through March 31, 198X. 



FIGURE THIRTY-TEREE: Sample Management Report 



REHABILITATION SERVICES, INC. 
Report to the Board 

Program Evaluation for the Month of March, 198X 

and 

The cumulative period January 1, 198X - March ;/l, 198X 



Program Effectiveness 

During the month of March, 80% of the clients who were 
closed from the vocational evaluation program did so w«th posi- 
tive vocational recommendations. This is 5% below our goal. For 
the period beginning 1/1/84 and ending 3/3/84, we maintained our 
goal of 85% for this objective. 

For those who did complete their program, their average 
program length was five weeks for both March and the cumula- 
tive period. This is one week longer than our goal of four 
weeks. 

Of those clients who completed their program in March with 
vocational recommendations, 88% of those recommendations were 
followed by their referring vocational counselor 60 days after 
exiting their program. This is 8% over our goal of 80%. For the 
cumulative period, our results were 76% or 4% under our goal. 



Who Was Served 

The major disability group for whom we provide services 
remains to be the mentally retarded with 40% of our case loads 
reporting this as the primary disability diagnosed at intake. 
Including the mentally ill (20%), 60% of our clients have some 



156 



ERLC 0J 



mentally related disability. The physically disabled population 
(30%) has shown an increase, apparently due to an increase of 
referrals from insurance companies. They have increased the 
number of injured workers who have entered our programs. The 
remaining clientele show disabilities of hearing impairment (2.5%), 
visual impairment (2.5%), learning disability (2.5%), and epilepsy 
(2.5%). Fifty percent of our clients are multiply and/or severely 
disabled. 

We continue to serve a large proportion of minority clients 
(50%), most of whom arc black. Fifty percent of those being 
referred have had no previous history of employment. This has 
remained true for 50-75% of our caseload for the past year. 
Thirty percent of cvaluccs have been previously institutionalized. 

Our major referral source continues to be the State Voca- 
tional Rehabilitation Office with 50% of our clients being referred 
from that source. However, we arc seeing more referrals from 
private insurance companies who are referring workers compensa- 
tion claimants. This month these clients comprised 20% of our 
client load. 



Rehabilitation Costs 

It cost the facility $6,284.00 to operate the vocational 
evaluation program for the month of March. The cost per client 
(8) for those who completed the evaluation program with positive 
vocational recommendations was, therefore, $786,00. The facility 
received! $4,713.00 in purchase of service fees from state and 
local sources to provide evaluation programming during the 
month, and $900.00 was received from insurance companies. This 
left a balance of $671.00 that will be balanced by revenues from 
internal sources. 

The total costs for the cumulative period were $18,751. 
This is $552.00 per client for these completing their program with 
positive vocational recommendations. Purchase of service from 
state and local sources amounted to $16,495 and $2,675 was 
received from insurance companies, leaving a net operating deficit 
of $1,019.00. 



Program Recommendations 

It is recommended that a closer look be taken at the rea- 
sons for our evaluations lasting an average of five weeks, or one 
week more than our goal of four weeks. We are receiving con- 
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tinued pressure from all purchase of service revenue sources to 
minimize the time that clients spend in the program. Possible 
solutions are: 

1. Evaluate the amount of time spent in the evalua- 
tion program per data gathered. Could some of 
the data be gathered through situational assessment 
methods? 

2. Examine what assessment tools and areas are 
taking the longest time and investigate alternative 
testing products now available on the market that 
would reduce testing time. 

3. Determine if evaluations are being as individually 
structured as is needed for certain client needs 
and referral sources or whether all clients are 
going through the same procedures, some of which 
may be unnecessary. 

It is recommended that since we are now serving more 
physically disabled clients than before, that we determine whether 
we have sufficient evaluative tools to access the needs of these 
clients. 

It is recommended that the facility further develop and 
market its services to workers' compensation claimants due to 
initial successes, increasing familiarity of this group by program 
staff, and insufficient financial support to operate the vocational 
evaluation program by governmental sources. An organized and 
structured program needs to be developed along with an ap- 
propriate brochure if the program is to be professionally mar- 
keted to insurance carriers and companies. 
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EXERCISE TWEIVE: Write a Management Report 



Using data below, write a management report that details the program's 
effectiveness, describes who was served, the rehabilitation costs incurred, 
and your recommendations for future operation (including decisions about 
program termination or expansion,) Include at leust one visual representa- 
tion of your findings. 

FIX)W PATH FOR REHABILITATION SERVICES, INC. 
Referral 



Refer for services? 



Refer internally? 




V 



Placement 



V 

© 
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EXERCISE TWELVE 
CONTINUES 



Place competitively? 



V 



Sheltered employment 




No 



No job 



END OF 19XX CLIENT TOTALS FOR REHABILITATION SERVICES, INC. 





Evaluation 


Training A 


Placement 


Total 


Referrals 


520 


52 


26 


598 


Served 


390 


224 


222 


468 


Self -terminate 


39 


68 


0 


107 


No benefits 


92 


68 


7 


167 


Placements 


23 


105 


145 


145 
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SELF-TEST: Chapter Eight 



make written reports more meaningful. 

2. Program evaluation reports are often called reports. 

3. When data analysis is discussed, levels and data 

should be reported as well as test and values. 

4. Program evaluation reports should include: program results by , 

program results by . , , a 

of results, and . 

5. Evaluation reports should be designed to help 

make about the facility and its programs. 
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SELF-TEST: Chapter Eight 
ANSWER KEY 



1. Graphic displays make written reports more meaningful, 

2. Program evaluation reports are often called management reports, 

3. When data analysis is discussed, significance levels and data means 
should be reported as well as test statistics and critical values, 

4. Program evaluation reports should include: program results by objec- 
tive* program results by client descriptors, a summary of results, and 
recommendations, 

5. Evaluation reports should be designed to help stakeholders make deci- 
sions about the facility and its programs. 
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Chapter Nine 
Develop a Plan to Implement Program Evaluation 



No program evaluation will provide long term input into the facility's 
planning and accountability systems without ongoing commitment from the 
board of directors and chief executive officer. 

Program evaluation, as defined by the Commission on Accreditation of 
Rehabilitation Facilities (CARF, 1985) is a systematic procedure for deter- 
mining the effectiveness and efficiency with which esults following reha- 
bilitation services are achieved by persons serve... Evaluation data col- 
lected regularly and/or continuously can provide a rich source of data to 
assess the facility's effectiveness and efficiency at providing rehabilitative 
services. Program evaluation data will keep the administrator appraised of 
the facility's status regarding the completion of its mission. In addition, 
these data will be used to identify and change client flew problems, result- 
ing in better client service. 

Program evaluation is also a tool for providing better services. In a 
program evaluation, administrators, accreditation agencies, and funding 
bodies examine the facility's ability to deliver the services they were or- 
ganized to provide. By regularly examining facility programs, data is pro- 
duced that is of great value to funding bodies. Essentially, program evalua- 
tion is an integral part of the service delivery system. 

The essential elements of program evaluation, according to CARF 
(1975), are: 

(1) A purpose statement. 

(2) A structuring of programs. 

(3) A system review mechanism. 

(4) Manager nt reports. 

(5) Goal stat ^ents for each program. 

(6) Admission criteria for each program. 

(7) A listing of services for each program. 

(8) A listing of persons served for each program. 

(9) Measurable objectives for each program. 
(10) Measurement criteria for each objective. 
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(11) Specification of who is measured for each objec- 
tive. 

(12) Specification of the time each measure is applied 
for each objective, 

(13) Specification of variables for each person served, 
including severity and barriers. 

(14) Specification of success criteria for each objective. 

(15) Specification of relative importance of each objec- 
tive. 

Note that the major elements of program evaluation include an element 
requiring system review (element number 3). Id. addition, many of the 
elements call for the creation of ongoing data collection systems (under- 
lined.) These data will be analyzed using many types of program evaluation 
research techniques, including systems analysis, experimental studies, case 
studies, and correlational research. Thus, ongoing program evaluation data 
collection systems support the activities of the entire facility. 



Create an Implementation Strategy 

The implementation of a program evaluation process into a rehabilita- 
tion facility must be carefully planned. Careful planning helps insure that 
the evaluation is relevant. It is also essential in order to counter the 
inevitable resistance that will be encountered from program staff. 

Once the facility has developed its program evaluation structure, a 
trial implementation period is begun. This trial is undertaken for the 
following reasons: 

1. To complete the development of all needed forms and docu- 
ments. 

2. To develop procedures for information flow, collection, 
tabulation, and reporting. 

3. To prepare a manual of operations. 

4. To train staff. 

5. To collect data. 

6. To generate initial reports. 

7. To critique and modify the system. 

A period of six months is recommended for the trial implementation 
period once data collection has begun. 
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FIGURE THIRTY-FOUR: A Sample Installation Plan 



STEP DESCRIPTION ASSIGNED STARTED COMPLETED 



1 


LXBCUt iv e c onus jl uxuexi u 








2 


Policy decision/reso- 
lution from governing 








3 


Appoint program eval- 
uation committee and 
poiooii responsijjiu xor 
developing the system 








4 


Develop written poli- 
cies t procedures , and 
system elemerts 








A 1 
H • ± 


Schedule regular meet** 
ing with key indi- 
viduals 








4.2 


Develop a written plan 
which includes all 
sysueui eienenus 








4.3 


Analyze current data 
for a clear definition 








5 


Establish a trial 








5.1 


uovoio^ rsporu jccjaIus 
and source documents 








5.2 


Develop system data 
flow chart 








5.3 


Merge data collection 
system into facility 
information system 








5.4 


Prepare and publish 
manuals and forms 








6 


Implement the system 








7 


Review and critique the 
system no less than 
six months after im- 
plementation 








8 


Analyze data and write 
management report 








9 


Monitor the system 






i 
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Resistance to Program Evaluation 



We have already seen that program evaluation serves many functions. 
While the improvement of services for program recipients is the most 
obvious and usual stated reason for engaging in program evaluation, it is 
also used as a management tool to monitor/evaluate program staff and to 
account for the function of program administrators themselves. The very 
term "evaluation" denotes a threat. If program participants perceive that 
an evaluation will be used for covert reasons, resistance will be present. 

Often the issue of program evaluation will be as basic as the philo- 
sophical difference between service providers and administrators. They 
stem from two approaches on how best to understand and help human 
beings. Practitioners generally believe that knowledge can best be gained 
through studies of individuals. This point of view is rooted in the human- 
istic-holistic philosophy which sees the greatest good in the individual. On 
the other hand, program evaluation is steeped in the analytical-mechanistic 
tradition which holds that an individual can only be understood as a whole, 
but states that the best way to perceive the whole is to analyze the parts. 

Other differences between service providers and those who would 
evaluate their services stem from job role perceptions. Practitioners have 
fixed schedules, administrators do not. Practitioners are deeply committed 
to individuals, administrators see their commitment attached to groups. 
Practitioners gain nothing immediately from evaluation, administrators do. 
It is not difficult to see why service staff are often suspicious of program 
evaluation. 

Cook and Cooper (1978) cited Borgatta (1966) for the suggestion that 
facility staff will irrationally discount program evaluation findings both 
before evaluations take place and after they are implemented. 

"Among the arguments that program-related staff use to 
discount evaluation findings suggestive of program change (before 
the evaluation takes place) are: 

1. program effects are long ranged, they can't be 
ascertained in six months, one year, two years, etc; 
or 

2. program effects are too diffuse and general to be 
measured by any specific measurement device; or 

3. no instrument or measurement device is sensitive 
enough to measure small but important program 
effects; or 
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4. the measurement process in and of itself disturbs the 
program processes involved.* 1 (p. 20) 

"Borgatta also suggested rationalization that might be used to 
discount Negative' evaluation findings after the fact; namely: 

1. Some of the persons exposed to the program made 
great improvement; 

2. some of the persons who most needed the program 
and would show the most benefits were in the con- 
trol group; and 

3. no difference on outcomes between the program and 
control group implies need for a more intensive pro- 
gram." (p.21) 



Steps in Countering Resistance 

There are several planning steps that the facility may engage to 
counter this resistance. 

First, support for the development and implementation of a program 
evaluation must be generated from the highest administrative levels of the 
organization. Second, the plan should be flexible, allowing methodology and 
data collection to be gathered in ways that are the most accepted by the 
service staff. Third, program staff should be involved in the program 
evaluation. Their involvement should begin with the planning stage. Staff 
involvement may be generated by: 

A. Meeting with small groups of service personnel to describe 
the need for program evaluation and the necessity for fol- 
lowing a research design. 

B. Encouraging suggestions on how the evaluation may best be 
carried out. 

C. Remaining neutral as to administration-staff conflicts. 

D. Pointing out the benefits the evaluation will have for staff, 
such as improvement in services, documentation of effort, 
increased professional growth, etc. 
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EXERCISE THIRTEEN: Develop an Installation Plan 



Using the form above (or a similar form that better meets your facil- 
ity's needs) create an installation plan for your facility. 
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SELF-TEST: Chapter Nine 



1. Program evaluation needs from the facility board and CEO 

to be successful. 

2. One major element of a program evaluation plan is a plan for imple- 
mentation that includes . 



3. Implementation of a program evaluation must be carefully planned to 
insure that the data collected , 



4. Preplanning also helps counter 



5. Trial evaluation programs: 

a. Insure development of , 

b. Insure development of . 

c. Allow time to prepare a of operations. 

d. Provide staff time. 

e. Improve data . 

f. Show stake-holders initial . 

g. Give time for . 



6. If program participants perceive that an evaluation will be used for 
, resistance will be present. 



7. Resistance to program evaluation may be countered by three planned 

processes: gaining from the highest 

levels, using staff accepted methods, and 

by involving staff in the . 



8. Staff involvement can be gained by: meeting with in 

the planning stages, encouraging on how to evalu- 
ate program*,, avoiding administration staff , and 

stressing the f that evaluation 

will have for the staff. 
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SELF-TEST: Chapter Nine 
ANSWER KEY 



1. Program evaluation needs commitment from the facility board and CEO 
to be successful. 

2. One major element of a program evaluation plan is a plan for imple- 
mentation that includes scheduled reviews. 

3. Implementation of a program evaluation must be carefully planned to 
insure that the data collected is relevant . 

4. Preplanning also helps counter resistance. 

5. Trial evaluation programs: 

a. Insure development of needed forms. 

b. Insure development of flow procedures. 

c. Allow time to prepare a manual of operations. 

d. Provide staff training time. 

e. Improve data projection estimates . 

f . Show stakeholders initial results. 

g. Give time for modifications. 

6. If program participants perceive that an evaluation will be used for 
covert activities* resistance will be present. 

7. Resistance to program evaluation may be countered by three planned 
processes: gaining full commitment from the highest levels, using staff 
accepted data collection methods, and by involving staff in the planning 
process . 

8. Staff involvement can be gained by: meeting with small groups in the 
planning stages, encouraging suggestions on how to evaluate programs, 
avoiding administration staff conflicts, and stressing the positive effects 
that evaluation will have for the staff. 
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Appendix A 
Definition of Terms 



Addictive Disorders 

Percent of clients who received treatment services for alcohol or 
drug addiction. 

Assistance 

Purpose of contact is to resolve or prevent a problem. Surveil- 
lance or casual contact is not included. 

Blinu 

Percent of clients that have less than 5/200 visual acuity. 
Cessation 

The treatment services that are designed to achieve goals or 
resolve problems arc no longer being provided. 

Clients 

Persons officially recorded as program starts. Normally a person 
becomes a client on the day they enter or begin receiving ser- 
vices from a program. 

Continuous 

Completion of 90 calendar days with one employer. 

Developmentally Disabled 

Polio, epileptic, cerebral palsy, and mentally retarded. 
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Emotional Illness 



Percent of clients hospitalized for emotional problems or who 
received services from *\ iiiental health treatment facility. 



Employment 

A minimum of thirty hours of work per week at the same loca- 
tion or for the same person when compensation is received for 
time and effort expended is considered full time employment. 
This excludes seasonal and temporary positions. 



Family 

One other person with whom the patient may live, such as a 
spouse, brother, sister, son, or daughter. Excluded are multiple 
family members in an extended family. 



Full-time Competitive 

At least 30 hours a week, minimum wage, non-seasonal and 
permanent. 

Hearing Impaired 

Documented in referral data or by referral from the State Com- 
mission for the Blind. 



High School Diploma 

G.E.D. is an equal as is graduation from special education. 



Income 

Includes revenues from entitlement monies (such as SSI, private 
insurance, income earned by the release of a family to work, and 
food stamps). Excludes lump sum payments such as legal settle- 
ment, and workmen's compensation. 
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Increase 



The entitlement income at 90 days minus the entitlement at 
admission. 

Mentally Retarded 

Percent of clients diagnosed as retarded, with an intelligence 
quotient (IQ) of 80 or below. 

Occupational Skill Training 

Specialized training in a particular areas of employment. Ob- 
tainable in an accredited vocational school or OJT program. 

On-the-job Training (OJT) 

Involves a written client contract, wages earned in training. The 
expectation is that the client will remain on the job after the 
completion of training. 

Orthopedic 

Any skeletal or related deformities. 

Other "Status 26* 

Any other successful client iase closure. 

Poor Job Performance 

Determined by application for services and on social history 
taken during intake period. 

Program Length 

The number of weeks between the first and last day of services 
which are paid by funding sources. 
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Severely Disabled 



A disability requiring multiple services over an extended period of 
time. 



Severely Handicapped 

Percent of clients identified by funding agencies as having se 
verely handicapping disability conditions. 



Severely Unemployed 

Percent of clients having no prior work experience. Percent of 
clients who were unemployed during the entire t ree month 
period just prior to program entry. 



Social/Recreational Experience 

These experiences can be achieved by a visit made by another 
family member or friend to the member (even for short periods 
of only five minutes.) They can also be achieved by attendance 
at an organized program of a social or recreational nature. 



Special Person 

Someone who is paid to provide assistance. 



Terminees 

All clients who have obtained benefits or clients who have com- 
pleted the competitive employment program and have not entered 
a benefit category wi. „n 60 days of service completion. 



Voluntary 

Partaking of activities not mandated by self-maintenance. 
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Appendix C 
Program Evaluation Forms 



All the forms used in this manual can be found in this appendix. You 
are free to copy these forms and alter them to make them useful for 
program evaluation in your facility. They are placed in the appendix in the 
order that they appear in the manual. 
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PROGRAM STAFF WORKSHEFT 
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PROGRAM STRUCTURE CHART 



Influei. sr 



J 



Influencer Influencer Influencer 



Mission Statement 



Adnission Criteria 



Progj 


ram A 


Prog] 


ram B 


Program C 










Program Goal 


Program Goal 


Program Goal 










Admission Criteria 


Admission Criteria 


Admission Criteria 










Services Provided 


Services Provided 


Services Provided 










Clients Served 


Clients Served 


Clients Served 
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PROGRAM STRUCTURE WORKSHEET 

LIST INFLUENCES! 

1. 

2. 

3. 

4. 

5. 

6. 

8. 

LIST MISSION STATEMENT: 



LIST FACILITY ADMISSION CRITERIA: 

1. 

2. 

3. 

4. 

5. 

6. — 

7. 

8. 
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PROGRAM STRUCTURE CHART 



INFLUENCER 



INFLUENCER 



INFLUENCER INFLUENCER 



1. 




2. 




3. 




4. 



















MISSION STATEMENT 



ADMISSION CRITERIA 

i 

1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
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PROGRAM ELEMENTS WORKSHEET 



PROGRAM TITLE; 



PROGRAM GOAL: 



ADMISSION CRITERIA; 



1. 
2. 
3. 
4. 



SERVICES PROVIDED; 



1. 
2. 
3. 
4. 
5. 
6. 
7. 



CLIENTS SERVED; 



PROGRAM ELEMENTS WORKSHEET CONTINUED 



PROGRAM OBJECTIVES: 

1. 

2. 

3. 

4. 

5. 

6. 

1. 
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PROGRAM ELEMENTS CHART 



PROGRAMS 



PROGRAM A 



RAM 



PROGRAM B 



PROGRAM C 
I 



PROGRAM GOAL PROGRAM GOAL PROGRAM GOAL 

J . L 



ADMISSION CRITERIA ADMISSION CRITERIA ADMISSION CRITERIA 
i ! , , ! 



1* 
2, 
3, 
4. 



SERVICES PROVIDED 
I 



1* 
2, 
3. 
4* 



CLIENTS SERVED 



1. 
2. 
3, 
4* 



1* 
2. 
3. 
4, 



1. 
2. 
3, 
4* 



SERVICES PROVIDED SERVICES PROVIDED 
J , , I 



1. 
2. 
3. 
4. 



1. 
2. 
3. 
4, 



CLIENTS SERVED 
I 



CLIENTS SERVEu 
I 



1. 
2. 
3. 
4* 



1- 
2. 
3. 
4. 
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EVALUATION MEASURES CHART 

MEASURES FOR THE 



PROGRAM 





(A) 


(B) 


(C) 


(D) 


(E) 


(F) 


# 


OBJECTIVES 


MEASURE 


APPLY 


TIME 


DATA 


GOAL 


1 














2 














3 














4 














5 
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CLIENT DATA SHEET 



A: GENERAL INFORMATION 



NAME- 



ADDRESS- 



REFERRAL SOURCE- 
DISABILITIES 



-SS#- 



-COUNSELOR- 



-AGE- 



-SEX- 



-PHONE- 



-PH#- 



PROGRAM MANAGER: 



PROGRAM STATUS: ACTIVE 



INTERRUPTED 
TO 



EXIT 



CLOSED SPAN 



PROGRAM TERMINATION: SELF-INITIATED- 



AGENCY INITIATE D- 



VOCATIONAL OBJECTIVES (OR EVALUATION PLANS) : 
DOT CODE- 



DOT CODE- 



DOT CODE- 



B: BENEFIT STATUS 



OUTCOME OBJECTIVES: 

OBJ* : 

OBJ if 



-EXIT- 



-CLOSED- 



-EXIim 



OBJfi : 

PROCESS OBJECTIVES: 

OBJl : 

OBJl > 



-EXIT- 



-EXIT- 



NON-BENEFIT ACHIEVER- 



-EXIT- 



-CLOSED- 



-CLOSED- 



-CLOSED- 



-CLOSED- 



VOCATIONAL OBJECTIVES: 
VOCATIONAL PLAN: 



RECOMMENDATIONS 



D: FOLLOW-UP/REFERRAL INFORMATION 



EMPLOYER/AGENCY- 
JOB TITLE 



-CONTACT- 



-DOT CODE- 



STARTING WAGE 



6 MONTH OTJ- 



-PH#- 



-12 MONTH OTJ- 
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CLIENT DATA SHEET CONTINUED 



E: CLIENT DESCRIPTORS 

PHYSICAL DISABILITY ... 

MENTAL ILLNESS 

MENTAL RETARDATION .... 
HEARING DISABILITY .... 

VISUAL DISABILITY 

DEVELOPMENTAL DISABILITY 
LEARNING DISABILITY ... 

EPILEPTIC 

MINORITY 

SUBSTANCE ABUSER 

PUBLIC OFFENDER 

MULTIPLE DISABILITY ... 

SEVERE DISABILITY 

INDUSTRIALLY INJURED >. 

FORMER CLIENT 

PUBLIC ASSISTANCE 

LESS THAN H.S. EDUCATION 
H.S. EDUCATION OR G.E.D. 
POST H.S. EDUCATION 
NO PREVIOUS EMPLOYMENT 

UNDER 21 

MALE 

FEMALE 

PREVIOUS INSTITUTION .. 

VR REFERRAL 

BLIND AGENCY REFERRAL . 
SCHOOL REFERRAL .... * 

JTPA 

WORKERS COMPENSATION .. 

SELF-REFERRAL 

DMHMR REFERRAL 

OTHER REFERRAL SOURCE . 
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PROGRAM EVALUATION RESULTS SUMMARY SHEET 



TIME period: 


1 


2 


3 


4 


5 


6 


7 


8 


9 


TOT 


1 OR 2 OBJECTIVES 
3 OR MORS OBJECTIVES 
NONBENEFIT ACHIEVER 
PROGRAM LENGTH {WAS) 






















JOB PLACEMENT 
WORK ADJUSTMENT 
SHELTERED EMPLOYMENT 
WORK ACTIVITIES 
SKILL TRAINING 
HOMEBOUND 
OUT REFERRAL 






















AGENCY TERMINATION 
SELF-TERMINATION 






















PHYSICAL DISABILITY 
MENTAL ILLNESS 
MENTAL RETARDATION 
HEARING DISABILITY 
VISUAL DISABILITY 
DEVELOPMENTAL DIS. 
LEARNING DISABILITY 
EPILEPTIC 
MINORITY 
SUBSTANCE ABUSER 
PUBLIC OFFENDER 
MULTIPLE DISABILITIES 
SEVERELY DISABLED 
INDUSTRIALLY INJURED 
FORMER CLIENT 
PUBLIC ASSISTANCE 
LESS THAN H.S. ED. 
H.S. ED. OR G.E.D. 
POST H.S. ED. 
NO PREVIOUS EMPLOY. 
UNDER 21 
MALE 
FEMALE 

PREV. INSTITUTION 

VR REFERRAL 

BLIND AGENCY REFERRAL 

SCHOOL REFERRAL 

JTPA 

WORKERS COMPENSATION 
SELF-REFERRAL 
DMHMR REFERRAL 
OTHER REFERRAL SOURCE 
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PROGRAM RESULTS BY CLIENT DESCRIPTORS 



PROGRAM: 

OBJECTIVE/MEASURE : 

PERIOD COVERED: 

NUMBER OF CASES CLOSED: 



Descriptors 
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PROGRAM RESULTS BY OBJECTIVE 



R 
A 
N 
K 


OBJECTIVES 


GOALS 


THIS PERIOD 


CUMULATIVE 


DATA 


RE- 
SULT 


VARI 
ANCE 


DATA 


RE- 
SULT 


VARI- 
ENCE 


1 


















2 


















3 


















4 


















5 
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FLOW SUMMARY CHART 





PROGRAM TYPE 




DATA TYPE 








TOTAL 


Referrals 
Served 

Self-terminate 
No benefits 
Placements 











LEAST SQUARES REGRESSION LINE GRAPH 
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TIME PERIOD ADJUSTMENT FORM 



Wages per Average Adjusted 

Mean Working working Days Mean 

Month Days Day Per Month Wages 

January r = X = 

February f = X = 

March 7 X 

April f X 

May * x 

June j. = X = 

July f X 

August f = x = 

September r = x = 

October f = X = 

November • = X = 

December r = x = 



SEASONAL ADJUSTMENT 



Average Seasonally 
Adjusted Adjusted Seasonal Adjusted Adjusted 



Month Wages Wages Index Wages Wages 

January f = X = 

February f = X = 

March i- X = 

April 7 = X 

May r = X 

June 5" = X = 

July J" X 

August r = X = 

September r = X = 

October r = X = 

November r = X - 

December • = X = 
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LEAST SQUARES FORECAST DATA 



X 

(TIME) 


X 1 


Y 

(WAGES) 


X'Y 


2 

X' 


1 










2 










3 










4 










5 










£ 

V 










7 










8 










9 










10 










11 










12 























N *= 




E X' 


E Y 


E X 1 Y 


2 

E X' 



MEAN PROJECTIONS 



Pro j ected 


Forecast 




Seasonal 




Month 


Mean 




Tndex 




January 




x 




m 


February 




X 




XM 


March 




X 




m 


April 




X 




K* 


May 




X 




as 


June 




X 




n 


July 




X 




m 


August 




X 




m 


September 




X 




m 


October 




X 




n 


November 




X 




m 


December 




X 




rx 



Seasonally 

Adjusted Running 
Mean Total 



SIMPLE CHI-SQUARE 



YES 



NO 



YES 



NO 



A 


B 


C 


D 



COMPLEX CHI-SQUARE DATA SHEET 



AREA 1 


1 


2 


3 


4 


5 


AREA 2 


6 


7 


8 


9 


10 


AREA 3 


11 


12 


13 


14 


15 
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COMPLEX CHI-SQUARE FORMULATION SHEET 



Observed Expected Observed minus Observed minus (O-E) divided 
Freq. Frequency Expected Freq. Expected Freq. by Expected 



O-E 



Squared 



( 0 - E ) 



Frequency 
( 0 - E ) 
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INSTALLATION PLAN 



STEP DESCRIPTION ASSIGNED STARTED COli^LETED 



1 


Executive commitment 








2 


Policy decision/reso- 
lution from governing 
body 








3 


Appoint program eval- 
uation committee and 
person responsible for 
developing the system 








4 


Develop written poli- 
cies, procedure^/ and 
system elements 








4.1 


Schedule regular meet- 
ing with key indi- 
viduals 








4.2 


Develop a written plan 
which includes all 
system elements 










Analyze current data 
fo£ a clear definition 
of needed changes 








5 


Establish a trial 
implementation date 








5,1 


Develop report forms 
and source documents 








5.2 


Develop system data 
flow chart 








5.3 


Merge data collection 
system into facility 
information system 




- - 




5.4 


Prepare and publish 
manuals and forms 








6 


Implement the system 








7 


Review and critique the 
system no less than 
six months after im- 
plementation 








8 


Analyze data and write 
management report 








9 


Monitor the system 
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Appendix D 
Formulas Used in Program Evaluation 



This appendix includes all the formulas mentioned in the text. These 
formulas are listed in the order that they appear in the document. 



Population Mean 



E (XI + X2 + Xi) 



Xi 



The sum of all the units of a population divided by the number of 
units in the population. 



Estimating a Sample Size 

When Problem is Expressed as a Proportion 



2 A A 

(Za/2) P (1-P) 

(Sample n = 

Size) 2 

ME 



A mathematical statistic (determined by using a table that takes into 
consideration the margin of error and probability of error) is first mul- 
tiplied by an estimate of the true proportion and that estimate minus one. 
The product of that multiplication is then divided by the square of the 
margin of error. 
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Reducing the Sample Size 
When n/N is Negligible 



(Sample size) 
n 

(Smaller n' = 



After the estimated sample size has been determined using the stan- 
dard formulas for both proportions and means, divide the sample size by the 
total population. If this number is greater than .05 you may reduce your 
sample size using the formula above. 

Divide the sample size by one plus the production of the division of 
the sample size by the total population. 



Estimating Sample Variance 

for Use in Determining Sample Sizes 

When Problem is Expressed in Means 



_ 2 
E (Y - Y) 
Sample Variance = 

N - 1 



The trial population mean is subtracted from each trial raw score. 
Each resulting figure is squared and all the products are summed. The 
summation is divided by the number of raw scores minus one. 
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Standard Deviation 



/ _ 2 

(SD) / E (Y - Y) 

Standard Deviation = / 

v n- 1 



To determine a group's standard deviation we must find the square 
root of the population variance. This is determined by subtracting the 
mean of the population from each score, squaring the products of the 
subtraction, summing the squared products, dividing the sum by the popula- 
tion total minus one, and finding the square root of the product. 



Z Scores 



Z = X - X 



SD 

Z scores are computed by determining the group mean, subtracting a 
raw score from the mean, and dividing the product by the group standard 
deviation. 
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T Scores 



T = (Z X 10) + 50 



Based on the Z score, T scores are determined by multiplying a Z 
score by 10 and adding 50 to the resulting product. 



Time Period Adjustment 

for Least Squares Forecasting 



RAW 

PERIOD 

(DATA) 



WORK 

PERIOD 

DAYS 



(DATA) 

PER 

DAYS 



(DATA) 

PER 

DAYS 



X 



AVE. 

DAYS PER 
PERIOD 



ADJUSTED 
MEAN 
(DATA) 
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Seasonal Adjustment 






for Least Squares Forecasting 






ADJUSTED • 


AVERAGE 


SEASONAL 


MEAN 


MEAN 


INDEX 


(DATA) 


(DATA) 


FIGURE 


SEASONAL 


AVERAGE 


SEASONALLY 


INDEX X 


MEAN 


ADJUSTED 


FIGURE 


(DATA) 


(DATA) 




Points on Line of Best Fit 






for Least Squares Forecasting 






E Y + X 1 x 


E X» Y 


= Point on line 


N 


2 


of best fit 




E X' 




First sum all data points and divide by the number of data points. To 


that figure add the statistic determined by manipulation of the independent 


variable. Multiply the resulting sum by the product of the following divi- 


sion: Multiply the independent variable statistic with each data point and 


add the resulting products. Then divide that summation by the sum of the 


squares of all the independent variable statistics. 
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Seasonalizing 

Least Squares Projections 



PROJECTED 
DATA X 



SEASONAL 
INDEX 



SEASONALLY 

ADJUSTED 

DATA 



Simple Chi-Square 
Test Statistic 



X 



N (AD-BC) 



(A+B) (C+D) (A+C) (B+D) 



Complex Chi-Square 
Test Statistic 



X = "E" 



( 0 - E ) 



E 



Expected frequencies are determined using the formula: 



E 



Sum of the Row X Sum of the column 
Total sample size (N) 
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